I am trying to make a web crawler which takes in a keyword and searches google to extract information about the keyword. The keyword contains application/product names. The aim of the crawler is to crawl google search results and get the following information about the application or product.

  1. Description
  2. Vendor Name
  3. URL

The solution I have right now is a rule based crawler, which takes in the keyword and based on some Xpaths and htmml classes gets the required information if available on the search page. The problem here is the solution is not scalable and it requires new rules to add when the requirement changes. I want a solution where i can train the bot to look for certain places on the search page and get the required information.

I do not know how to collect the data for this, how to label the dataset. What should the dataset look like, html pages?

Below is the part of the search page which the bot should be able to retrieve on its own: enter image description here

As you can see in the above image the information present in the right container is what I want the bot to extract.

Any direction, help is appreciated. Thanks in advance.

  • $\begingroup$ The right container is essentially a summary of the Wikipedia page for the proudct. Do you mean that's what you're interested in? If yes, you can download the full Wikipedia database and directly access the required page. $\endgroup$ – Erwan 2 days ago