201
$\begingroup$

One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level analysis. Looking at the other people's effort, especially other questions on this site, it appears that many people in this field are doing somewhat repetitive work. For example analyzing tweets, facebook posts, Wikipedia articles etc. is a part of a lot of big data problems.

Some of these data sets are accessible using public APIs provided by the provider site, but usually, some valuable information or metrics are missing from these APIs and everyone has to do the same analyses again and again. For example, although clustering users may depend on different use cases and selection of features, but having a base clustering of Twitter/Facebook users can be useful in many Big Data applications, which is neither provided by the API nor available publicly in independent data sets.

Is there any index or publicly available data set hosting site containing valuable data sets that can be reused in solving other big data problems? I mean something like GitHub (or a group of sites/public datasets or at least a comprehensive listing) for the data science. If not, what are the reasons for not having such a platform for data science? The commercial value of data, need to frequently update data sets, ...? Can we not have an open-source model for sharing data sets devised for data scientists?

Improve this question
$\endgroup$
9

35 Answers 35

Reset to default
1
2
3
$\begingroup$

Quite late to the party, but I believe the following might be helpful: http://data.worldbank.org/indicator
https://www.quandl.com/

Improve this answer
$\endgroup$
2
$\begingroup$

This question already has so many answers but I guess it will be very useful for machine-learning practitioners to use so many CSV files that are here.

Improve this answer
$\endgroup$
2
$\begingroup$

These are the most used ones:

  1. Kaggle: The Home of Data Science & Machine Learning. Kaggle helps you learn, work, and play.
  2. Govt. Repository Data: USCG Marine Casualty and Pollution Data
  3. Quandl: Quandl is a marketplace for financial, economic and alternative data delivered in modern formats for today's analysts, including Python, Excel, Matlab, R, and via our API.
  4. UCI WEB: Classification - Regression - Adult - Abalone - Life Sciences
Improve this answer
$\endgroup$
2
$\begingroup$

Adding on the publicly available BigQuery datasets where you can find:

Improve this answer
$\endgroup$
1
$\begingroup$

Now google also provides a dataset search for researchers!!

https://datasetsearch.research.google.com/

It gives you link to all the other sites. It's like one search to search them all!!

Improve this answer
$\endgroup$
1
2

Not the answer you're looking for? Browse other questions tagged or ask your own question.