Build a web scraper to retrieve specific website information on a regular basis

Ukończone Opublikowano 5 lat temu Płatność przy odbiorze
Ukończone Płatność przy odbiorze

Overview

For this project several bits of information must be gathered from a website and saved into a MySQL database. The information type is always the same but the parameters are different.

In most cases the website has APIs for retrieving the data and this should be the preferred way of getting the data.

Please see attached PDF for the exact data required.

Requirements

- Project needs to be created in Java.

- Application needs to use the Selenium Framework and Chrome/Firefox in either window mode or headless mode.

- All calls must be parameterised and should work for all Leagues, Teams, Players, Matches.

- APIs from the target website must be used only, no scraping of HTML. In some cases the data is embedded in the code using JSON, it is acceptable to retrieve the data from Javascript variables or using Javascript calls if APIs are not available.

- API endpoints seem to be available only after visiting the main pages, so some protection exists, possibly based on cookies. So this needs to be circumvented.

- Data will be saved in a MySQL database with tables per each dataset. Eg. Players, Teams, Leagues, Matches.

- IDs must be used and preferred as opposed to Strings. IDs from the target website must be used as opposed to locally created auto-increment IDs.

- For each ID stored in the database an associated table must exist with the String value. Eg. Player ID – Player Name.

- So tables need to be normalised.

Java JavaScript MySQL Pozyskiwanie danych z Internetu Testowanie serwisów WWW

Numer ID Projektu: #18194755

O projekcie

5 ofert Zdalny projekt Aktywny 5 lat temu

Przyznany użytkownikowi:

p4logics

Hello Sir, We are expert in web scrapping using java and Python. we will write automation script which you I will setup at you local system or server. or you can run it manually on daily/weekly basis. please ping me Więcej

£80 GBP w ciągu 20 dni
(67 ocen)
6.6

5 freelancerów złożyło ofertę za £109 w tym projekcie

Yknox

Good Day Sir! I read your post. But I have something to ask you. Your idea is amazing and it will change the world! I am a magic talented developer in your skill. If you wanna be the success, hire me I am looking Więcej

£50 GBP w ciągu 10 dni
(671 Oceny)
9.0
catamarin

Hi there, I see you're looking for an automation engineer, with excellent skills in selenium and java, to help you with your website scraper project. Do you already have the MYSQL database created? Have you already Więcej

£77 GBP w ciągu 15 dni
(21 Oceny)
4.6
lysilab

Hi Sir, I am Lavish Karankar having vast experience in Automation of jobs using Python and JAVA language. I have gone through your provided description, I would like to help you on this as I have done this task in my Więcej

£88 GBP w ciągu 10 dni
(4 Oceny)
4.2