A program which can fetch data into MongoDB

Ukończone Opublikowano 7 lat temu Płatność przy odbiorze
Ukończone Płatność przy odbiorze

We need some one to build a program which can fetch data from website into MongoDB for us. The program should:

1. work for most of the websites. It's NOT a focused crawler.

2. visit, scan and fetch the content of all the pages under the domain URL we input. It should enter the process by domain URL. For example, if we input "[url removed, login to view]", it should start the process with "[url removed, login to view]"

3. remove the web code , only keep the content, and save them according to our requirements

4. same the real link(absolute path) of the picture, not the relative path

Storage requirement:

1. The URLs we input are with a company name. like <[url removed, login to view]><xxx company>. The company name and the website address should be saved as the first level of the data collection. So that we can know which url the content came from

2. The data stored in MongoDB should have the same structure as they were in the web [url removed, login to view] like using F12 to check the elements of the web page

Additional requirement:

Removing the content of header and side slider. Since they are not the major content, they are not necessary for us. Will pay more if some one can make it.

Java NoSQL Couch oraz Mongo PHP Python Architektura oprogramowania

Numer ID Projektu: #10424379

O projekcie

18 ofert Zdalny projekt Aktywny 7 lat temu

Przyznany użytkownikowi:

gcsekhar002

hi The previous data scrapig works on this website are from secure websites ctrac, tripadvisor and expedia I'm a freelance developer from 3 years and as a freelancer I develop automation works like filling forms, fet Więcej

$500 USD w ciągu 5 dni
(12 ocen)
4.1

18 freelancerów złożyło ofertę za $722 w tym projekcie

gopalvora

Hi I have gone through the details of your project and we find it well within our capabilities. I offer a wide range of services, including Web design, PHP/MySQL web application development, Open sources like Joo Więcej

$721 USD w ciągu 20 dni
(393 Oceny)
8.0
TenStar718

Hello, and thanks for the opportunity to bid on your project. https://www.freelancer.com/u/TenStar718.html I am an expert in many different area’s of web and mobile applications based on the following languages: W Więcej

$526 USD w ciągu 10 dni
(130 Oceny)
7.3
mobileappdevin

We have a good amount of experience in web scraping using Python,Django and nodejs. This is our latest project on web scraping using python: Scraping using Python: Electronics Parts Intelligence Processing ePr Więcej

$1666 USD w ciągu 30 dni
(21 Oceny)
6.6
akhila27

Hello, Before you select a part time developer from here, take a look at our portfolio: fugacode.com. If you like what you see, contact us. That's all. "Why hire part time college students? when you can hire prof Więcej

$555 USD w ciągu 10 dni
(20 Oceny)
6.3
lillysoft

Sir i am really interested in your project . sir i am an expert of software and web development. i have already developed many web and windows applications and some are similar to your project . you can check my port Więcej

$555 USD w ciągu 3 dni
(15 Oceny)
5.0
amrkh

Hi, I need a few sample URLs and output for the html extraction. This should be relatively simple. I am proficient in all automation technologies related to web such as Selenium or HtmlAgility. Waiting for samples f Więcej

$555 USD w ciągu 10 dni
(23 Oceny)
4.5
winnow1

Everything is clear except "remove the web code , only keep the content, and save them according to our requirements" Let me share my understanding of it: 1. Remove all HTML tags, Javascript etc. and keep only the Więcej

$1000 USD w ciągu 20 dni
(1 Ocena)
3.8
MohanKumar28

Hi, I have good experience in scrapping web data using php and jquery ajax. Have gone through the requirements. We can do this. Please share additional details. Thanks, Mohan

$333 USD w ciągu 4 dni
(14 Oceny)
4.0
HealthyCoder

Hello Sir, Being a Software Engineer i can do your job easily, i have 3 years experience with Web application development, come to chat for detail conversation about your project. Regards Sibghat Ullah

$500 USD w ciągu 10 dni
(14 Oceny)
4.0
thamtrinh

Hello, My name is Tham and I'm developer in Vietnam who specialize in creating and developing websites like Windows, Linux, Mac OS. I have much experience in web development. I have cooperated to develop successfully Więcej

$500 USD w ciągu 15 dni
(0 Oceny)
0.0
techminds4

Dear Prospect Hiring Manager. Thank you for giving me a chance to bid on your project. i am a serious bidder here and i have already worked on a similar project before and can deliver as u have mentioned I have c Więcej

$555 USD w ciągu 10 dni
(0 Oceny)
0.0
prithvirajkdm91

A proposal has not yet been provided

$777 USD w ciągu 10 dni
(0 Oceny)
0.0