can somebody provide help me to decide on what's a reasonable price for scraping blogs/forums like the following using python scrapy :
Every post (Date, text, user, title, categories) is supposed to be saved as an item and then into a MongoDB.
The offer I got is 30$ per blog/forum. Is that a fair price?
Thank you very much,
There isn't a standard price for web scraping since every website present a different set of challenges when it comes to scraping data from its pages. Some websites are easier to scrape while others aren't. The presence of JS, Ajax, Flash, CAPTCHA etc makes scraping data all the more difficult.
Also, you need to consider the security measures some sites apply to prevent bots and scripts from scraping their pages. In such cases, the developer needs to adopt advanced measures for bypassing such filters.
So the pricing can only be ascertained once the website that needs to be scraped has been reviewed and analyzed thoroughly, taking all the above points in consideration.
Hope this helps!