[Lestrades.com] Nao 尚

  • I'm Share-Locked
  • Administrator
  • Posts: 1,366
  • Trades: 219
Help test Lestrade's!
« on March 9th, 2017, 12:54 AM »
And another topic which I hope will be a bit more lively. ;)

For now, I'm trying to see if my wishlist/tradable scraper is good enough, and I'm asking for volunteers to:
- open their wishlist & tradables
- open the same on barter.vg (using the same order, IIRC by time added)
- compare both. You shouldn't have to bother about whether all entries are here (only the unlinked ones will be missing), I'm mostly looking to see if all tags are correctly shown.

The reason I'm asking is because last week I noticed I'd forgotten to import the 'extra' tag for wishlist items (i.e. items you already have in your tradables/library but that you want more than one copy of.)
So I'm guessing it's possible I forgot something else, although unlikely. (The 'extra' tag was shown differently in Barter, i.e. with a '+' sign instead of the wishlist's place of origin.)

joaoarena

  • Jr. Member
  • Posts: 99
  • Trades: 26
Re: Help test Lestrade's!
« Reply #1, on March 17th, 2017, 01:46 PM »
I see all my tags there. The only issue is, when we have 2 games with the same name and store tag that are not aggregated on barter (ie listed twice because I added duplicates directly from bundles), it overwrites the first and shows only one copy.
This does not happens when there is a different store tag, or it was originally x2 and on.
My real wishlist is on steam.
My wishlist here is for trading purposes.
I like to trade for games I do not own based on Want/Have Ratio, rather than other approaches.

/u/humpaaa

  • Full Member
  • ▶ ¦ Yep
  • Posts: 189
  • Trades: 10
Re: Help test Lestrade's!
« Reply #2, on March 17th, 2017, 04:38 PM »
Well, i can tell you something that works:

Recently, i deleted ~100 games from my account.
Barter did not update my Library, LT has the correct amount of games

Can confirm unlinked games are missing.

I barely used tags, apart from tier indicators. These have been importe correctly.


[Lestrades.com] Nao 尚

  • I'm Share-Locked
  • Administrator
  • Posts: 1,366
  • Trades: 219
Re: Help test Lestrade's!
« Reply #3, on March 17th, 2017, 05:54 PM »
I'm surprised, normally your removed games should still be there, but with a 'removed' flag next to them... (I tested on castle of illusion when it got removed from everyone who bought a key.)

As for barter, I think steam list imports broke forever when login broke. As if their api key got revoked, actually... (but steam would probably tell us their key was revoked.)
Re: Help test Lestrade's!
« Reply #4, on March 17th, 2017, 07:38 PM »
Note: I've figured out why I was removing (game + store) duplicates. That's because that's how I know which item to delete when I'm updating the database with new tradables from Barter... *except* I've actually never got around to doing that: I used to say "people will have their tradables on LT and will be able to import from Barter in case they keep using it", except by now it'd be totally silly to keep using Barter once the offer system is in place, ahah... So, no reason to remove such games. So I'll be changing my code, don't bother!
I'll be doing big changes internally to accomodate for the offer system though-- 90% of my code accounts for people having Barter accounts and addresses them by Barter ID, rather than LT ID. There is a big chance that when I enable the trade system, there will be a few hours where no one will be able to make any matches at all, or the site may even be down (or crash) while I'm downloading Barter data one last time before shutting down the Barter list scraper.

It's a complicated step, so please bear with me. The offer system is 'working' (barely...... But it's fucking working and it's the first time in a week I've been able to create an offer and show it in my offer list), but I need to completely sever ties with Barter as a result. It shouldn't be a problem because there are nearly 700 users here, I think..? So I'll be importing data from these 700 users. You'll no longer be getting 'inactive' user matches, although I'll end up doing something similar based on last online date. (But it's not implemented yet.)

Real life update: we're Friday night here, my kid got back from the nanny's and I'll be off for a few hours. Once he's asleep I'll get to spend another couple hours on the site, but it's unlikely I'll setup the big import for tonight. So I'm planning it for tomorrow instead. Fingers crossed!!

Akylen

  • Full Member
  • Posts: 215
  • Trades: 60
Re: Help test Lestrade's!
« Reply #5, on March 18th, 2017, 01:36 AM »
Nice stuff Lestrade. Yeah, it would be good once the offer system is completed to just switch off Barter. Very low activity due to the recent problems with login.

Let me know if you need more help with the testing.

[Lestrades.com] Nao 尚

  • I'm Share-Locked
  • Administrator
  • Posts: 1,366
  • Trades: 219
Re: Help test Lestrade's!
« Reply #6, on March 18th, 2017, 10:48 AM »
Yeah I will.

Okay, I've modified my internals so that you can now sync your duplicate items. I think it will work, although it didn't change anything for me since I already fixed my dupes yesterday. ^^

Please tell me guys if this fixes your numbers. This'll be the last opportunity to check for Barter discrepancies before I do 'the' final scrape.
Posted: March 18th, 2017, 09:25 AM

BTW, I stand corrected, Barter finally imported my wishlist data yesterday.

Caviezel

  • Newbie
  • Posts: 17
  • Trades: 3
Re: Help test Lestrade's!
« Reply #7, on March 18th, 2017, 02:39 PM »
Sorting tradables seems to be working randomly?

Will there be Wants and Haves sorting?

[Lestrades.com] Nao 尚

  • I'm Share-Locked
  • Administrator
  • Posts: 1,366
  • Trades: 219
Re: Help test Lestrade's!
« Reply #8, on March 18th, 2017, 04:55 PM »
Quote from Donpeste on March 18th, 2017, 02:39 PM
Sorting tradables seems to be working randomly?
I... don't think so?
It's not remembered across pages or refreshes, if that's what you mean. Not a priority right now. Also, the original sort order is usually 'recent items', but for the third party trader you're offering to (which, I know, not available etc.), it's currently a random order, because meh.
Quote from Donpeste on March 18th, 2017, 02:39 PM
Will there be Wants and Haves sorting?
I don't know, is it needed? I'll add them if I feel like I need them.
In the meantime you can use the Ratio sort. If I add Haves & Wants, I may reserve Ratio sorting for premium users so... be careful what you ask for :niark:

joaoarena

  • Jr. Member
  • Posts: 99
  • Trades: 26
Re: Help test Lestrade's!
« Reply #9, on March 19th, 2017, 01:44 AM »
Quote from Lestrade on March 18th, 2017, 10:48 AM
Please tell me guys if this fixes your numbers. This'll be the last opportunity to check for Barter discrepancies before I do 'the' final scrape.
Posted: March 18th, 2017, 09:25 AM


That's right, fixed my tradables and my Library (have not sincronyzed barter lybrary, so I dont know, maybe library data comes from steam?)
My real wishlist is on steam.
My wishlist here is for trading purposes.
I like to trade for games I do not own based on Want/Have Ratio, rather than other approaches.

[Lestrades.com] Nao 尚

  • I'm Share-Locked
  • Administrator
  • Posts: 1,366
  • Trades: 219
Re: Help test Lestrade's!
« Reply #10, on March 19th, 2017, 10:27 AM »
Good, good!

Yes, library data is coming from Steam exclusively, contrary to the rest. Importing wishlists & co already takes an hour per scrape, adding libraries would probably add an extra hour, all for nothing because right now the website doesn't make use of anything.

I'll keep in mind that maybe I should import Barter libraries as a basis for the new database, after which the regular Steam library import will take over on a regular basis.

Gosh, I need to have a to-do-list somewhere...

joaoarena

  • Jr. Member
  • Posts: 99
  • Trades: 26
Re: Help test Lestrade's!
« Reply #11, on March 19th, 2017, 12:03 PM »
Quote from Lestrade on March 19th, 2017, 10:27 AM
...  I'll keep in mind that maybe I should import Barter libraries as a basis for the new database, after which the regular Steam library import will take over on a regular basis.

Gosh, I need to have a to-do-list somewhere...
Importing new games from steam is great.
What about new bundles?
They have to be manually added. Will you implement that?
My real wishlist is on steam.
My wishlist here is for trading purposes.
I like to trade for games I do not own based on Want/Have Ratio, rather than other approaches.

Forum Conventions

  • Newbie
  • Posts: 37
  • Trades: 0

[Lestrades.com] Nao 尚

  • I'm Share-Locked
  • Administrator
  • Posts: 1,366
  • Trades: 219
Re: Help test Lestrade's!
« Reply #13, on March 19th, 2017, 04:19 PM »
Yes, I wrote a scraper last year, scraped all of their bundles until something like January, and then proceeded to fix a lot of them (early bundles had different ways of being handled). Then I got bored by the job and didn't bother to automate processing new bundles. Also, I wasn't sure whether or not I should import pending bundles, since it makes adding new bundles faster, but also more prone to errors + duplicates. Plus, the scraper doesn't support re-scraping already visited bundles-- so it's a problem for bundles like the current Humble Jumbo.

I estimate that the bundle situation won't move much for now, but I may try and fix the scraper to at least get it up to date with recent bundles. And maybe do that manually once in a while. It's not a complicated job, I just need to jump back into it.