Data extraction using crawlers

Question

I have a rather simple data scraping task, but my knowledge of web scraping is limited. I have a excel file containing the names of 500 cities in a column, and I'd like to find their distance from a fixed city, say Montreal. I have found this website which gives the desired distance (in both km and miles).

For each of these 500 cities, I'd like to read the name in the excel file, enter it in the "to" box, set the "from" field to "Montreal", press on the "Find" button (or Enter), extract the distance in km, and store the result in a vector.

Is there any source which walks you through these steps in Python, R, or even an online service?

Answer 1

I think there are few different ways to handle such a task. You can try creating a python script using pandas for example for extracting the names of the cities and then use Selenium for navigating and finally save the results back to file using pandas.

For that you need a basic HTML understanding to find the HTML elements that needed to be handled both for inserting your data into form and for collecting the results after each city is inserted.

Here is a source explains the basics of finding elements with Selenium

Answer 2

One of the best modern books on web-scraping is Mojolicious web clients by brian d foy because you will need to know more than just how to fetch a web page. You will need to know how to locate that information on the page and Mojo has great support for CSS selectors in addition to HTTP headers and authentication.

Make sure you read Chapter 1, section Be Nice to Servers. A bot running amok is a good way to get banned from a site.

A second option is to see if there is a page with all the data on it so that you can avoid having to crawl and just parse.