Generic Crawler (PHP & Code Igniter)


I need a crawler that knows to identify products from any site/page (the site owner knows about it!):

1. You get the URL of the homepage, parse it with PHP ganon, find all the images of the homepage and their links that should be the SRC of the closest A parent backwards

2. You group the found images by one of these options IN THIS ORDER (the first that is applicable):

a) Grouped by [url removed, login to view] (or DOM OBJECT POSITION)

b) Grouped by [url removed, login to view] + [url removed, login to view] if there are into the IMG TAG (don't try to reach width and height in PHP at this stage)

c) Grouped by number of directory levels (/) into [url removed, login to view] && number of directory levels (/) into [url removed, login to view]

d) Grouped by margin of similar_text of [url removed, login to view] && margin of similar_text of [url removed, login to view]

e) Grouped by [url removed, login to view] + [url removed, login to view] after reaching width and height of the images via PHP

3. You start with the group of images that have the higher number of images, and you look backwards for the price and the name of the item after checking that the name looks like [url removed, login to view] (in case of doubt open the item page and take the TITLE but it should be rare!)

4. When you find 3 images of the same image's group that have url, name and price, that's all! You will stop the headaches and start to search in the homepage and then in all the sites ONLY the CSS classes that you have just found now

5. For every product, you find the tags from CONCAT([url removed, login to view], '', [url removed, login to view], '', NAME) - a word that appears at least twice is a tag...

You release first homepage results ASAP and IN PARALLEL you run a script that takes other links from the sitemap or from this analyze where you already have a ganon object that will give you all the links of the page easily and quickly

The crawler must be multi thread thanks to Multi Thread CURL of PHP and it will work of course for the most of the site = millions sites, so it should be 1000% generic!

This task is very complex and relevant only for PHP engineers that wants to progress in their career...

Good luck!

Umiejętności: Codeigniter, MySQL, PHP, Architektura oprogramowania, Pozyskiwanie danych z Internetu

Zobacz więcej: mp3 crawler source code php, php crawler product, php code web crawler database, crawler code php, ebay product list code php, php code website crawler, php crawler code, website crawler code php, simple web crawler script code php, product rating reviews open source code php, code php add product, code igniter interface php, php crawler html tags, code web crawler php, php crawler tags contents, add product code php, security code php, mail code php post check, video url embedd code php, convertir code php php

O pracodawcy:
( 5 ocen ) Petach Tikva, Israel

Numer ID Projektu: #12550540

54 freelancerów złożyło ofertę na kwotę $586 do tego projektu


Hello, and thanks for the opportunity to bid on your project. I am an expert in many different area’s of web and mobile applications based on the following languages: W Więcej

$526 USD w ciągu 10 dni
(155 Oceny)

Hi I have gone through the details of your project and we find it well within our capabilities. I offer a wide range of services, including Web design, PHP/MySQL web application development, Open sources like Joo Więcej

$567 USD w ciągu 14 dni
(425 Oceny)

Hello, Hope you are doing well !! php codeigniter Experts!! I have gone through your requirements. Lets discuss more over Chat box, So i can define the best suitable time frame to get it finished at the earliest Więcej

$750 USD w ciągu 10 dni
(156 Oceny)

Dear sir, I am scraping expert, I have did too many scraping projects, please check my reviews then you will know. Can you tell me more details? then I will provide example data/script for you. Thanks, Więcej

$557 USD w ciągu 6 dni
(253 Oceny)

Dear Client, Greetings from Flowgica technologies, I have experience with these skills. We do have similar experience doing something similar to yours therefore I am looking forward to discuss more about more details a Więcej

$525 USD w ciągu 12 dni
(40 Oceny)

Dear Sir/Madam, We are in top 25 company in , we made lots of system and as well also earned more then 1000000+ USD and give us one chance then you will know our level of expertise and quality wor Więcej

$722 USD w ciągu 20 dni
(164 Oceny)

Hey Friend, Myself having has 10 + years experience of web site development. Expert in Custom programming in php / Mysql / Jquery / Wordpress/ Joomla / OSC / Magento /OpenCart / Android / Mobile development / R Więcej

$580 USD w ciągu 10 dni
(198 Oceny)

======EXPERTS HERE READY TO START===== Hi, we have a creative and logical web development Professional Team with great skills specialized in WORDPRESS, CSS, HTML, PHP, ONLINE STORES, ECOMMERCE , GRAPHIC DESIGNING , et Więcej

$250 USD w ciągu 5 dni
(131 Oceny)

Hello, my name is Olya. I represent Ukrainian IT Company «Webbook». We provide website design and web and app development services for organizations, public and government institutions, company or private web-pages. Więcej

$600 USD w ciągu 20 dni
(60 Oceny)

Hello, I understood the initial scope of this project. Although i want to discuss further this job in order to prepare the final concept for this project. After Complete discussion over the call or in chat, i wi Więcej

$773 USD w ciągu 10 dni
(50 Oceny)

Hello, Thanks for giving me the opportunity to discuss your project. I am a professional PHP developer with an experience of more than 8 years in creating and customizing PHP based websites, themes and plugins/m Więcej

$750 USD w ciągu 10 dni
(61 Oceny)

Dear client, Thank you for allowing me to post bid on your project. We have an expert team of professional blog, web application, responsive and mobile website developers in [login to view URL] have many satisfied clients Więcej

$700 USD w ciągu 10 dni
(89 Oceny)

Hi there, My name is Momin. I’ve read your brief and can see that I’d like to do this task. My team has 7 years experience designing and developing. I would approach your project by starting with wireframes Więcej

$555 USD w ciągu 10 dni
(133 Oceny)

Dear Client , We gone through project description We are Really interested to do work with you , We are Web & Software development company having more than 8 year experience in Software development we totally co Więcej

$555 USD w ciągu 10 dni
(154 Oceny)

Hello ! We want to discuss about your project as we have experience in it. What Differentiates us from the other freelancers : # Experience of more than 5 years in Unity 3D, Xcode, CoCoa 2d, Phonegap and major Więcej

$1000 USD w ciągu 10 dni
(56 Oceny)

I have gone through the requirement and still need some more brief of the project. I have more than 6 years of experience in skills like Ecommerce, Enable Shopping cart, Payment and API Implement/Integration, Javas Więcej

$526 USD w ciągu 15 dni
(33 Oceny)

Hello, I have gone through your outsourced requirements "Generic Crawler (PHP & Code Igniter)" and have analyzed that I have right skills (Codeigniter, MySQL, PHP, Software Architecture, Web Scraping) to execu Więcej

$586 USD w ciągu 365 dni
(102 Oceny)

Hi There, Ready To Start now. I have 4+ years experience in PHP as well as Wordpress, Magento,OpenCart, Prestashop, OsCommerce….and also in responsive CSS3, bootstrap, html5 . etc You can check our work http://www Więcej

$333 USD w ciągu 2 dni
(140 Oceny)

Hi! I am a experienced developer myself and leads a team of 3 designers and 5 developers. We are specializing in designing and building websites. If you want your website to be more professional, then we are the righ Więcej

$650 USD w ciągu 10 dni
(95 Oceny)

Hi, We are go through your brief and can see that you would like to build a WEBSITE. We make a detail review of your given description and understand it very well. We will build with all features as you required. Więcej

$1000 USD w ciągu 18 dni
(52 Oceny)