Forum Conventions

  • Newbie
  • Posts: 37
  • Trades: 0
Barter Offer Archive
« on March 18th, 2017, 01:01 AM »Last edited on March 22nd, 2017, 07:47 AM
Sayonara, Barter
http://sb.antigravities.net/

I figured I'd post this here in case anyone's interested: in the event that another meteor hits Barter.vg, I've started to keep an archive of every offer made, since nobody else has done it. The project is called "Sayonara, Barter.vg", and a crappy frontend (that I plan on improving at some point) to the collected data is here (keep in mind it downloads ~25MB in compressed data, mobile users beware!). I've written a new semi-buggy backend that you should use, here. It is also very far from complete.

If you'd like to conduct your own science experiment with the data, I've made the source to the scraper open and the data files available. Here are offers.lz (huge, ~17mb - an archive of all of the offers), users.lz (big, ~5mb - an archive of basic user information), and items.lz (tiny, ~200kb - an archive of every item that's been mentioned in an offer). A full scrape took approximately 20 hours, but your results may vary. I plan to re-scrape about every week.

Extracting the data to JSON is relatively easy if you have access to Node.js. Simply
Code: [Select]
npm install lz-string
and then
Code: [Select]
var fs = require("fs");
var lz = require("lz-string");

["offers", "users", "items"].forEach(function(v){
  fs.writeFileSync(v + ".json", lz.decompressFromEncodedURIComponent(fs.readFileSync(v + ".json").toString()));
});

// Ta-da, you should now have ~125M of JSON in the current directory. Isn't file compression amazing?

Have fun! If anyone notices anything wrong with the scraped data, please let me know so I can fix it. Also let me know if you're doing anything interesting with it, I'd love to hear about that!

Onion Wizard

  • Newbie
  • Posts: 1
  • Trades: 0

Forum Conventions

  • Newbie
  • Posts: 37
  • Trades: 0
Re: Barter Offer Archive
« Reply #2, on March 21st, 2017, 08:39 AM »
The scraper now pulls comments, and I've updated the code to the latest version on GitHub.

KeyG

  • Newbie
  • Posts: 4
  • Trades: 7
Re: Barter Offer Archive
« Reply #3, on March 21st, 2017, 02:00 PM »
can somebody write sometimes right instructions? No notion like this should function.

Forum Conventions

  • Newbie
  • Posts: 37
  • Trades: 0
Re: Barter Offer Archive
« Reply #4, on March 21st, 2017, 03:58 PM »Last edited on March 21st, 2017, 04:03 PM
Instructions for... using the scraper? I don't think it's really necessary, the reason I open sourced it is so that I could show how I was getting the data and the schema of the database itself. You should consider using the 3 .lz files that I linked above if you want to do your own experiments (which are compressed with lz-string and can be decompressed by using the above method), or the web site if you just want to view offers.

In other news, I'm currently doing a rescrape that should be finished by end of day ET today. It will contain offer comments.
Re: Barter Offer Archive
« Reply #5, on March 22nd, 2017, 04:07 AM »Last edited on March 22nd, 2017, 05:35 AM
The scrape just finished in approx. 19 hours. The new data will be up on the server once it compiles, and I am currently almost finished with the new frontend.

Fun fact: people have sent 885 offers since the scrape last completed.

Edit: The new archive has been uploaded and has increased by 5MB in size.

joaoarena

  • Jr. Member
  • Posts: 99
  • Trades: 26
Re: Barter Offer Archive
« Reply #6, on March 22nd, 2017, 11:04 AM »
Quote from Alex 🍕s Onion Wizard on March 22nd, 2017, 04:07 AM
Fun fact: people have sent 885 offers since the scrape last completed.
Me included, I'm using the matches system here to see who is still on Barter, and send offers to only those. Completed some already.
My real wishlist is on steam.
My wishlist here is for trading purposes.
I like to trade for games I do not own based on Want/Have Ratio, rather than other approaches.

Forum Conventions

  • Newbie
  • Posts: 37
  • Trades: 0

Quantum

  • Newbie
  • Posts: 30
  • Trades: 4
Re: Barter Offer Archive
« Reply #8, on March 22nd, 2017, 11:45 PM »
Barter was at 1300 active users when I first noticed the login problem. It's down to 774 now (which is still a pretty decent amount really).

Forum Conventions

  • Newbie
  • Posts: 37
  • Trades: 0
Re: Barter Offer Archive
« Reply #9, on March 25th, 2017, 04:42 PM »
Starting another scrape today which should be done by tomorrow morning ET.
Re: Barter Offer Archive
« Reply #10, on March 26th, 2017, 03:18 PM »
Scrape is done. Somehow lost offers, -5854. Will compile and have up by end of day
Re: Barter Offer Archive
« Reply #11, on March 26th, 2017, 05:54 PM »
Uploading the new database right now. If you're scraping my site, please don't, I'd be more than happy to provide you with the databases.