I recently started working as a data scientist and I am starting a web scraping and NLP project using Python. The idea is to create a program that searches for public information on the company's clients. These information can come from various sources: annual reports, income statements, articles.... I will have to deal with two types of formats: HTML and PDFs. For now I will focus on retrieving the revenue of the company. After a month of research and tests, I realized a few things: - NLP techniques are too slow to be used on annuals reports The first step of the project will be the following:
Search for the annual report and scrape the HTML code: so far I managed to get all the google results and I'm using Beautifulsoup to get the HTML code. However I can't quite get the revenue of the company because each website has its own HTML structure. I first decided to focus on extracting tables (the goal is to find the company's income statement) but I realized that HTML tables are often used for layout (even if it's a bad practice). I can't rely on css selectors as I need to keep it as generic as possible. How can I achieve it?