**Quick Summary:** I need a web crawler to go through search result pages and crawl the sites for email addresses.
**Detailed Description:** I will have a list of keywords in a .txt file, each keyword will be on a new line. I want the program to take the keyword, go to google, msn, and yahoo (the exact search query links will be at the bottom of this post). It will then go into each search result and crawl the ENTIRE site and store any email addresses it finds. It will do this for ALL the search results.
The results will be stored in a .txt with the naming like this: [login to view URL]
I would like a progress notification telling me off the bat the number of results for the keyword for the specific search engine, and then a counter as to which site result is being crawled, and then a counter for the number of email addresses it has found.
**Example:
**keyword: vector marketing canada
google results: 156,000
yahoo: 4,040,000
msn: 814,000
This means that the crawler will have crawled 5,010,000 sites and all the pages within each site.
The results would be stored in: [login to view URL]
**Search Queries:**
[login to view URL]
[login to view URL]
[login to view URL];_ylt=1?p=vector+marketing+canada
Please be aware that there is a "+" (plus sign) where there are spaces in queries.
**Note:** Please make sure you have a clear understanding of how to crawl for email addresses, as some email addresses have multiple subdomains:
ex: [login to view URL]@[login to view URL]
**Lastly. Please INCLUDE your estimated time for product delivery as well as your chosen programming language. Once I have chosen you as a bidder I will confirm this deadline. Please make sure you can meet deadlines.
**