Hi,
I need a bot made that will scrape a forum and keep a database of it locally. It must have a function that I can hit and it will crawl/scrape the forum again for new posts and update the database. The forum uses simple machine forum platform. I only want to scrape 1 category of the forum.
It must also show me stats of the threads that it scraped/crawled. I want to be able to see how many pages per day each thread is growing and sort it by highest % on top. I want to be able to see how many unique users a thread has, again sort it with the highest number on top.
I would also like to select on stats to calculate stats on threads that are X days (min) to X days (max) only.
Proxies:
Software must allow proxys.
I would like to have an option to load from a list (.txt) or from a url. I would like an option to reload list every x mins. I want an option to use an ip only X times in X mins. I want an option to retry on errors (connection timeouts), I would like to set the connection timeout as well.
I will give url to the person I hire. Please be very efficient at coding desktop applications.
Also bot must use multiple threads.
I also expect the source code when project is done.
thanks.
Hi sir,
I am scraping expert, I have did too many similar projects, please check my feedback then you will know.
Can you tell me more details? then I will provide demo data for you.
Thanks,
Kimi