Missing bundles
[Groupees] Bundle pre-orders »

Akylen

  • Full Member
  • Posts: 218
  • Trades: 61

[Lestrades.com] Nao 尚

  • I'm Share-Locked
  • Administrator
  • Posts: 1,396
  • Trades: 222
Re: Missing bundles
« Reply #1, on August 15th, 2019, 06:32 PM »
All bundles are scraped twice an hour from ITAD, which back in 2017 was a good solution. BVG's bundle page is now mostly better maintained, but I'm not too fond of the idea of scraping bundles from them as well. I like that Lestrade's allows me to skip checking out ITAD and sometimes have some bundles that bvg doesn't track.
The only way to add this bundle to Lestrade's would be to manually add it to ITAD, which is doable, but their interface isn't exactly great.
I'm still considering other possibilities, e.g. having a manual tool for the audience here. But it means more work for me, ahah...
Re: Missing bundles
« Reply #2, on August 16th, 2019, 11:57 AM »
I can confirm that ITAD is now preventing Lestrade's from acquiring its latest bundles, even though they're user-submitted (some were even fixed by me... Does it mean I can't reuse my own work? :P)
I don't know why the guy would bother banning us... I mean I'm only sending two requests per hour... :-/
I'm trying to find an alternative solution to get the bundle list, otherwise I'll just have to call it quits and remove ITAD scraping. (I was getting bored of fixing all of the broken bundles anyway :P)

Akylen

  • Full Member
  • Posts: 218
  • Trades: 61
Re: Missing bundles
« Reply #3, on August 16th, 2019, 12:47 PM »
Oh that's unfortunate. Why would he do that. Doesn't barter scrape ITAD too?
Perhaps many of the people who contribute to ITAD also do it for Barter and Lestrades (yourself is a good example). Isn't that a shoot in the foot?

There is always a possibility of scraping reddit

[Lestrades.com] Nao 尚

  • I'm Share-Locked
  • Administrator
  • Posts: 1,396
  • Trades: 222
Re: Missing bundles
« Reply #4, on August 18th, 2019, 02:04 AM »
Quote from Akylen on August 16th, 2019, 12:47 PM
Oh that's unfortunate. Why would he do that.
I don't know. Maybe some other website was scraping irresponsibly and without telling, I don't know. Maybe we all get punished for it. Or maybe he's just trying to plug Augmented Steam (even though it has nothing to do with LT's...)
Quote
Doesn't barter scrape ITAD too?
No, it's doesn't. One of the reasons I decided to scrape ITAD instead of just reusing the BVG database for this, is that at the time, there weren't a lot of older bundles in their database, and ITAD had them all. Things have changed: in two years most of the old (interesting) bundles were added, and the new ones are just as relevant. (Which doesn't mean it's mistake-free, but it's nice to have both.)
Quote
Perhaps many of the people who contribute to ITAD also do it for Barter and Lestrades (yourself is a good example). Isn't that a shoot in the foot?
LT's is only edited by yours truly. I could technically give editing powers to others, but... I didn't get any requests..?
AFAIK, the couple of volunteers who fill out the BVG bundle list do not do the same for ITAD.
Quote
There is always a possibility of scraping reddit
That one lacks information. The Steamgifts topics are better for that, but even then, AFAIK the announcement message structure isn't flawless (evolves over time, no absolute consensus, etc.)

Akylen

  • Full Member
  • Posts: 218
  • Trades: 61
Re: Missing bundles
« Reply #6, on August 21st, 2019, 12:54 PM »
One thing I noticed is when I compare the Library to Barter, Lestrades actually adds the bundle packages/collections, which is super cool.
However I notice that some of the items are being incorrectly recognised as not having there. For ex:
https://lestrades.com/game/1038/
https://lestrades.com/game/17362/
https://lestrades.com/game/4595/
The actions are not persistent, and when ignore is pressed it reloads the page and back to square zero. Once I do the comparison again, it shows up again. Same when I press to remove item from library, like demo games. Which is a bit silly.

There is no way I can remove this game from its game page
https://lestrades.com/game/39365/

For some items, there is no way to add it on Barter :/
https://barter.vg/i/57696/

Concluding... although these problems are exclusive to Lestrades, it is a good thing. The database is more complete and has every little thing that we own. Thanks for putting up the work on that!

[Lestrades.com] Nao 尚

  • I'm Share-Locked
  • Administrator
  • Posts: 1,396
  • Trades: 222
Re: Missing bundles
« Reply #7, on August 22nd, 2019, 11:15 AM »
Games that can be added/removed manually from library:
- DLCs
- Pending games (i.e. games that aren't dropping cards yet and not showing up in your public library yet.)
- Steam packages (because they don't show up in public libraries, but they're discovered by the user API, just like both above.)
I don't think Barter does the distinction for pending games. Maybe you can add them through one of the JavaScript extensions, though.

Games that you mentioned are:
- One is an episode (you can buy episodes on Steam but you can't get a Steam key for them, so they're not tradable... And usually, you buy the whole series so that's the thing that's potentially tradable.)
- Config titles, i.e. internal stuff that isn't tradable. One is a Mac version of GTA, which is included in the game when you install it, it's got its own app ID so it's listed at Lestrade's, but if you want to get that, you need to install the parent game. (Or sometimes sibling game, depending how I set up the relationship in the database.)
Generally, anything that gets its own app ID but is installed as part of something else or not found on the Steam store (retail-only keys, level editors for a parent game, private server software, demo, trailer...) won't be tradable on Lestrade's. And thus, can't be removed from your library manually either. I'm flexible on changing details, though! But generally speaking, LT's/BVG's database is really full of stuff that shouldn't even be in there... Which is why they're not discoverable by default.. (You have to select 'junk' in the search type.)
For instance, whenever Steam does a publisher sale weekend, it creates a store page which internally has an app ID, and you can access it through that app ID. This ends up in their huge database and the Steam API returns the type as 'advertising', but... not always. (App types are done manually at Steam, sometimes I guess there are human errors. Which I then manually fix on Lestrade's. But then again there's also human error at Lestrade's, ah ah.)

BTW I suppose you've tried adding games via the 'Add DLCs' button, too..? Sometimes it helps where other solutions don't.

Akylen

  • Full Member
  • Posts: 218
  • Trades: 61
Re: Missing bundles
« Reply #8, on August 22nd, 2019, 11:37 AM »
Yeah, I've used that before.
But are the buttons functioning as you expect? If only I could ignore these, because I honestly don't care about the junk.

[Lestrades.com] Nao 尚

  • I'm Share-Locked
  • Administrator
  • Posts: 1,396
  • Trades: 222
Re: Missing bundles
« Reply #9, on August 25th, 2019, 01:24 AM »
I don't know, I've tailored the system to my own use, which as close as possible to 'logical' as I could imagine.
Junk needs to stay in the database, in case stuff was flagged as junk incorrectly.

PS: I took the liberty to remove tons of bundles from the active bundle list, the StarDock-published ones... They're mostly not bundles, just a glorified sale. Probably GMG has a sake in StarDock or the other way around... I don't see the point of considering a $50 bundle to be called a bundle in the first place.

Akylen

  • Full Member
  • Posts: 218
  • Trades: 61
Re: Missing bundles
« Reply #10, on August 25th, 2019, 02:39 PM »
I didn't mean to suggest to remove them from the DB. They are useful on their own right. But most of the stuff I would expect to be able to ignore permanently from the the comparison.

[Groupees] Bundle pre-orders »