Lestrade's

The Trader Club => The Trader's Clerk => Topic started by: Forum Conventions on March 18th, 2017, 01:01 AM

Title: Barter Offer Archive
Post by: Forum Conventions on March 18th, 2017, 01:01 AM
Sayonara, Barter
http://sb.antigravities.net/

I figured I'd post this here in case anyone's interested: in the event that another meteor(http://steamcommunity.com/groups/bartervg/discussions/0/133259227518441895/) hits Barter.vg, I've started to keep an archive of every offer made, since nobody else has done it. The project is called "Sayonara, Barter.vg", and a crappy frontend (that I plan on improving at some point) to the collected data is here(http://sayonara-barter.antigravities.net/) (keep in mind it downloads ~25MB in compressed data, mobile users beware!). I've written a new semi-buggy backend that you should use, here(http://sb.antigravities.net/). It is also very far from complete.

If you'd like to conduct your own science experiment with the data, I've made the source to the scraper open(https://github.com/antigravities/sayonara) and the data files available. Here are offers.lz(http://sayonara-barter.antigravities.net/offers.lz) (huge, ~17mb - an archive of all of the offers), users.lz(http://sayonara-barter.antigravities.net/users.lz) (big, ~5mb - an archive of basic user information), and items.lz(http://sayonara-barter.antigravities.net/items.lz) (tiny, ~200kb - an archive of every item that's been mentioned in an offer). A full scrape took approximately 20 hours, but your results may vary. I plan to re-scrape about every week.

Extracting the data to JSON is relatively easy if you have access to Node.js. Simply
Code: [Select]
npm install lz-string
and then
Code: [Select]
var fs = require("fs");
var lz = require("lz-string");

["offers", "users", "items"].forEach(function(v){
  fs.writeFileSync(v + ".json", lz.decompressFromEncodedURIComponent(fs.readFileSync(v + ".json").toString()));
});

// Ta-da, you should now have ~125M of JSON in the current directory. Isn't file compression amazing?

Have fun! If anyone notices anything wrong with the scraped data, please let me know so I can fix it. Also let me know if you're doing anything interesting with it, I'd love to hear about that!
Title: Re: Barter Offer Archive
Post by: Onion Wizard on March 18th, 2017, 05:37 AM
Good work.
. :thanks:
Title: Re: Barter Offer Archive
Post by: Forum Conventions on March 21st, 2017, 08:39 AM
The scraper now pulls comments, and I've updated the code to the latest version on GitHub.
Title: Re: Barter Offer Archive
Post by: KeyG on March 21st, 2017, 02:00 PM
can somebody write sometimes right instructions? No notion like this should function.
Title: Re: Barter Offer Archive
Post by: Forum Conventions on March 21st, 2017, 03:58 PM
Instructions for... using the scraper? I don't think it's really necessary, the reason I open sourced it is so that I could show how I was getting the data and the schema of the database itself. You should consider using the 3 .lz files that I linked above if you want to do your own experiments (which are compressed with lz-string and can be decompressed by using the above method), or the web site if you just want to view offers.

In other news, I'm currently doing a rescrape that should be finished by end of day ET today. It will contain offer comments.
Title: Re: Barter Offer Archive
Post by: Forum Conventions on March 22nd, 2017, 04:07 AM
The scrape just finished in approx. 19 hours. The new data will be up on the server once it compiles, and I am currently almost finished with the new frontend.

Fun fact: people have sent 885 offers since the scrape last completed.

Edit: The new archive has been uploaded and has increased by 5MB in size.
Title: Re: Barter Offer Archive
Post by: joaoarena on March 22nd, 2017, 11:04 AM
Quote from Alex 🍕s Onion Wizard on March 22nd, 2017, 04:07 AM
Fun fact: people have sent 885 offers since the scrape last completed.
Me included, I'm using the matches system here to see who is still on Barter, and send offers to only those. Completed some already.
Title: Re: Barter Offer Archive
Post by: Forum Conventions on March 22nd, 2017, 10:37 PM
New interface is done, minus searching, so I'll be taking the old one down soontm. Next scrape is in 3 days, then I'll start doing it weekly.

http://sb.antigravities.net/
Title: Re: Barter Offer Archive
Post by: Quantum on March 22nd, 2017, 11:45 PM
Barter was at 1300 active users when I first noticed the login problem. It's down to 774 now (which is still a pretty decent amount really).
Title: Re: Barter Offer Archive
Post by: Forum Conventions on March 25th, 2017, 04:42 PM
Starting another scrape today which should be done by tomorrow morning ET.
Title: Re: Barter Offer Archive
Post by: Forum Conventions on March 26th, 2017, 03:18 PM
Scrape is done. Somehow lost offers, -5854. Will compile and have up by end of day
Title: Re: Barter Offer Archive
Post by: Forum Conventions on March 26th, 2017, 05:54 PM
Uploading the new database right now. If you're scraping my site, please don't, I'd be more than happy to provide you with the databases.