I would like to pull out the title and year of the all articles published in BBC since 2010 based on some keyword search for instance [MBA, LBS, Harvard...... and so on]. How can I do this using python? BeautifulSoup takes a link and extract some text from the link rather than searching the relevant links by itself based on some keywords.
if you want to just pull news articles from BBC news, then you could use their API.
they provide client libraries in different languages (including python)
You can search for articles with any combination of the following criteria:
Keyword or phrase. Eg: find all articles containing the word 'Microsoft'.
Date published. Eg: find all articles published yesterday.
- Source name. Eg: find all articles by 'TechCrunch'.
- Source domain name. Eg: find all articles published on nytimes.com.
- Language. Eg: find all articles written in English.
I believe the first one is what you are looking for.
You need to use a search engine for this. The official way is to pay for API calls in Google. You can also try scraping Google web page, but they do not like it and do stuff to prevent it. So you can only call one page every second (time interval may be different). As a second step you could scrape a page with the article if google does not provide enough information.