Crawler should work the way similar to <[login to view URL]>. Well I don't want it to display URL after URL manually. I want it to read the URLs from the file, crawl the site and write discovered data into the file. There will be 2-3 parameters (tags) that need to be handled. Data is currently in CSV format and need to be stored into the CSV, but this issue is negotiable.
Preferance - tested script. Let me know your if you have one.
Please let me know:
A. Platform.
B. Speed & Condition of script. It's very important. I need to process large amount of urls. So please make sure computer or server will not be crashe with the large data.
C. Possibility of customization. I don't think that it will be needed, because it's basically one time activity. So I don't need it to be too fancy
## Deliverables
1) Complete and fully-functional bug-free working program(s) in executable form as well as complete source code of all work done.
2) Deliverables must be in ready-to-run condition and be tested on performance issue with the large amount of data.
## Platform
Linux / PHP / MySQL