hi! I need a hand in configuration Xvfb and watir to scrape headless website in AJAX using RHEL7 or Amazon Linux AMI.
(Removed by [url removed, login to view] Admin) with screen sharing to configure my EC2 instance to start working. For experienced people work for less than 1h I estimate.
I have built an app on Rails that scrapes one website twice a day to fetch some data using Nokogiri
Now I wanted to add one more source (another website) that is built in Ajax and Nokogiri is not able to fetch the data and since my app is on AWS I need some headless solution.
So I’m experimenting with Headless gem along with Watir in Rails and Xvfb in RHEL7 to simulate browser - the issue is - always when I trying to use Watir I’m getting an error on server: Selenium::WebDriver::Error::WebDriverError: Unable to find Mozilla geckodriver. Same error getting when trying with chrome.
Chrome and firefox and Xvfb seem to be properly installed.
On my local server Watir is triggering web browser and open in new window properly. The problem is with doing the same using headless solution.
fix the problem above only to start scraping by using headless Watir - nothing more.