So, I have to start thinking about the topic of my final project in a data science master's degree (business oriented, although I can choose any unrelated field) and one of the requirements is to mine and use data that has not yet been analysed in the academic research environment. I would prefer to avoid the typical scrape of data from twitter or other common scraping sources of information. I would really appreciate if you could give me some ideas or direction on how to find an accesible source of data which also does not require too much time to get information from.

Thanks a lot for the help!


1 Answer 1


If it's business oriented, there are many "business Wikipedia" type websites that have lots of data presented in the same format on each page, which will make them a lot simpler to scrape. For example Yahoo Finance for stock data finance.yahoo.com

You can use the BeautifulSoup library executed with a Python script locally to set-up a simple HTML page scraping script given the URL, and then just get it to loop through some set of different page URLs to get all the info you need.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.