Anything related to the actual scraping should go in this topic.
What's scraping, by the way..? It's just a word we use for the process of one website's script browsing some other website(s) and 'spidering' (stealing?) their data for their own use. It's what I've been doing on Barter to import user tradables. It's what Barter has been doing to import Steam game data.
At some point in the future I'll switch to importing data from Steam directly, rather than using Barter.
This will have a few advantages:
- Will get more up-to-date data for recent games.
- I won't have to deal with stupid Barter hickups.
- I can more logically support adding non-Steam games manually to the database.
But these pale in comparison with the drawbacks:
- Riskier, because Valve may not like it. (Even if I'm going to be as soft as possible, e.g. maybe updating only recent games, older games less frequently, and very old games very rarely, unless a user visits the Steam game page and reports something changed.)
- I have to rewrite the whole damn thing! Damn! Thing! No, thing isn't a curse word.
- I'll have to determine what exactly I'm importing. For instance, Barter imports movies and things like that. is that REALLY necessary..?! This feels like a waste of time and space to me. If some bundle ends up having a movie key (like Groupees does from time to time), we can add those individually to the database. If someone really wants a movie in their wishlist, err... I don't know what to do. See what I mean, when I'm saying I don't know what I should import.
- And yes, what about demos and free games..? Will the games always be free? Even if I clearly indicate on the game page that the game is free and shouldn't be traded, it's still a problem. If a game was previously commercial and is now free, should it be removed from the database, upsetting people who have a copy up for trade? (Some people like collecting those as Steam gifts... I think.)
You have 2 hours!
(No, you have more actually. I'm just taking a break from coding and I don't know what to post.)
What's scraping, by the way..? It's just a word we use for the process of one website's script browsing some other website(s) and 'spidering' (stealing?) their data for their own use. It's what I've been doing on Barter to import user tradables. It's what Barter has been doing to import Steam game data.
At some point in the future I'll switch to importing data from Steam directly, rather than using Barter.
This will have a few advantages:
- Will get more up-to-date data for recent games.
- I won't have to deal with stupid Barter hickups.
- I can more logically support adding non-Steam games manually to the database.
But these pale in comparison with the drawbacks:
- Riskier, because Valve may not like it. (Even if I'm going to be as soft as possible, e.g. maybe updating only recent games, older games less frequently, and very old games very rarely, unless a user visits the Steam game page and reports something changed.)
- I have to rewrite the whole damn thing! Damn! Thing! No, thing isn't a curse word.
- I'll have to determine what exactly I'm importing. For instance, Barter imports movies and things like that. is that REALLY necessary..?! This feels like a waste of time and space to me. If some bundle ends up having a movie key (like Groupees does from time to time), we can add those individually to the database. If someone really wants a movie in their wishlist, err... I don't know what to do. See what I mean, when I'm saying I don't know what I should import.
- And yes, what about demos and free games..? Will the games always be free? Even if I clearly indicate on the game page that the game is free and shouldn't be traded, it's still a problem. If a game was previously commercial and is now free, should it be removed from the database, upsetting people who have a copy up for trade? (Some people like collecting those as Steam gifts... I think.)
You have 2 hours!
(No, you have more actually. I'm just taking a break from coding and I don't know what to post.)
