I am trying to make a web crawler which takes in a keyword and searches google to extract information about the keyword. The keyword contains application/product names. The aim of the crawler is to crawl google search results and get the following information about the application or product.
- Vendor Name
The solution I have right now is a rule based crawler, which takes in the keyword and based on some Xpaths and htmml classes gets the required information if available on the search page. The problem here is the solution is not scalable and it requires new rules to add when the requirement changes. I want a solution where i can train the bot to look for certain places on the search page and get the required information.
I do not know how to collect the data for this, how to label the dataset. What should the dataset look like, html pages?
As you can see in the above image the information present in the right container is what I want the bot to extract.
Any direction, help is appreciated. Thanks in advance.