Longterm Builder of Data-Mining Engine Required

We require a genius-level server-side software developer to build a data-mining engine.

This engine will:

- be a constantly growing cluster of single scripts.

- utilize a constantly evolving data-mining library that the developer will create and continually revise to make each script function efficiently.

- enable each script to be responsible for mining data from one website.

- have a complete, secure, web-based index of all scripts to be managed, reviewed and scheduled (we will provide the user interface).

- record vital statistics as to the data it collects, successes, failures, and complete history.

- be cloud powered.

This project is a minimum 5 year project. You will be paid per data-mining script, and each script will be negotiated based on its complexity. We expect the compensation per script to range from $15 to $120 each.

The following responsibilities will be yours, and will not be compensated for independently, but will be part of the agreement:

1. Your scripts must all be committed to the engine's GitHub repo directly from the server via SSH, once its output and functionality is approved.

2. You must develop an alert system that advises if something has failed with a script, i.e. the website it was mining changed structure, or went offline.

3. You will be responsible for building a script that compresses and databases the data that is mined, in a manner that allows for rapid querying.

You will require an extremely analytical mind, and should be the type of developer that enjoys algorithms, and complex mathematical scripting.

This is a minimum 5 year contract and our goal for this engine is to have it scraping tens of thousands of websites, each script on its own schedule. There is a lot of money to be made for the right person, but this person will be highly skilled, highly reliable, highly determined, and highly creative in their ways of problem solving.

!!! ATTENTION !!!

There is no list of websites that you could possibly send us that will cause us to choose you over someone else. This job will be awarded to the developer who has read and understands the complexity and the potential of this engine, and explains not only why they would be the best at building it, but also why they WANT to be the one to build it.

This job is not for someone who will lose interest, someone who loses power or internet regularly, gets sick regularly, or has family problems regularly. We are quick to fire when we hear these things, as they hurt longterm projects.

If before you know whether you can do the job, you need to ask what type of technology the websites have that you'll be scraping, then you can't do the job. Because we'll be using this engine to scrape so many websites that you'll likely come across everything.

The budget means nothing. This project will likely compensate the developer 5 - 6 figures USD over time. There will be full negotiation before awarding the contract.

Good luck!

Algorytmy Eksploracja danych Administrowanie bazami danych Machine Learning (ML) Architektura oprogramowania

Numer ID Projektu: #4184527

O projekcie