202
$\begingroup$

One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level analysis. Looking at the other people's effort, especially other questions on this site, it appears that many people in this field are doing somewhat repetitive work. For example analyzing tweets, facebook posts, Wikipedia articles etc. is a part of a lot of big data problems.

Some of these data sets are accessible using public APIs provided by the provider site, but usually, some valuable information or metrics are missing from these APIs and everyone has to do the same analyses again and again. For example, although clustering users may depend on different use cases and selection of features, but having a base clustering of Twitter/Facebook users can be useful in many Big Data applications, which is neither provided by the API nor available publicly in independent data sets.

Is there any index or publicly available data set hosting site containing valuable data sets that can be reused in solving other big data problems? I mean something like GitHub (or a group of sites/public datasets or at least a comprehensive listing) for the data science. If not, what are the reasons for not having such a platform for data science? The commercial value of data, need to frequently update data sets, ...? Can we not have an open-source model for sharing data sets devised for data scientists?

Improve this question
$\endgroup$
9

35 Answers 35

Reset to default
1
2
112
$\begingroup$

There is, in fact, a very reasonable list of publicly-available datasets, supported by different enterprises/sources.

Some of them are below:

Now, two considerations on your question. First one, regarding policies of database sharing. From personal experience, there are some databases that can't be made publicly available, either for involving privacy restraints (as for some social network information) or for concerning government information (like health system databases).

Another point concerns the usage/application of the dataset. Although some bases can be reprocessed to suit the needs of the application, it would be great to have some nice organization of the datasets by purpose. The taxonomy should involve social graph analysis, itemset mining, classification, and lots of other research areas there may be.

Improve this answer
$\endgroup$
77
$\begingroup$

Update:

Kaggle.com, a home of modern data science & machine learning enthusiasts:), opened it's own repository of the data sets.

In addition to the listed sources.

Some social network data sets:

There are plenty of sources listed at Stats SE:

Improve this answer
$\endgroup$
45
$\begingroup$

There are many openly available data sets, one many people often overlook is data.gov. As mentioned previously Freebase is great, so are all the examples posted by @Rubens

Improve this answer
$\endgroup$
39
$\begingroup$

Freebase is a free community driven database that spans many interesting topics and contains about 2,5 billion facts in machine readable format. It is also have good API to perform data queries.

Here is another compiled list of open data sets

Improve this answer
$\endgroup$
1
  • $\begingroup$ Freebase is closing down and its database will move to Wikidata soon. $\endgroup$
    – cynddl
    Commented Dec 17, 2014 at 14:39
27
$\begingroup$

For time series data in particular, Quandl is an excellent resource -- an easily browsable directory of (mostly) clean time series.

One of their coolest features is open-data stock prices -- i.e. financial data that can be edited wiki-style, and isn't encumbered by licensing.

Improve this answer
$\endgroup$
23
$\begingroup$

Enigma is a repository of public available datasets. Its free plan offers public data search, with 10k API calls per month. Not all public databases are listed, but the list is enough for common cases.

I used it for academic research and it saved me a lot of time.

Another interesting source of data is the @unitedstates project, containing data and tools to collect them, about the United States (members of Congress, geographic shapes…).

Improve this answer
$\endgroup$
21
$\begingroup$

I would like to point to The Open Data Census. It is an initiative of the Open Knowledge Foundation based on contributions from open data advocates and experts around the world.

The value of Open data Census is open, community driven, and systematic effort to collect and update the database of open datasets globally on country and, in some cases, like U.S., on city level.

Also, it presents an opportunity to compare different countries and cities on in selected areas of interest.

Improve this answer
$\endgroup$
19
$\begingroup$

There is also another resource provided by The Guardian, the British Daily on their website. The datasets published by the Guardian Datablog are all hosted. Datasets related to Football Premier League Clubs' accounts, Inflation and GDP details of UK, Grammy awards data etc. The datasets are available at

Some more resources. Some of the datasets are in R format or R commads exist for directly importing data to R.

Improve this answer
$\endgroup$
18
$\begingroup$

Custom Google Search

You can use the Custom Google Search for datasets:

Google Custom Search: Datasets

It includes 230 sources and meta-sources of datasets, including all mentioned in this question. Please, feel free to exclude .gov and any other websites from results by adding " -.gov" or " -site.com" to the search line. Other Google Search Operators work.

Don't hesitate to contact me if you have ideas what websites to add.

IOGDS

The following service categorizes more than 1,000,000 public datasets:

IOGDS: International Open Government Dataset Search

Improve this answer
$\endgroup$
2
  • $\begingroup$ What are the parameters for the custom search link you provided? Does it search in a list of websites, keywords, etc.? $\endgroup$
    – Amir Ali Akbari
    Commented Dec 5, 2014 at 7:42
  • $\begingroup$ @AmirAliAkbari It searches through sources like Data.gov, Quandl, and other major data warehouses. $\endgroup$
    – Anton Tarasenko
    Commented Dec 5, 2014 at 12:39
17
$\begingroup$

Late answer, but here is an eclectic list of 100+ Interesting Data Sets

The blog post is fun and easy to read through (I have no affiliation). It's worth to scan through, and to scrape a few from the top:

  • Last words of every Texas inmate executed since 1984

  • 10,000 annotated images of cats

  • 2.2 million chess matches

Improve this answer
$\endgroup$
16
$\begingroup$

I've found this link in Data Science Central with a list of free datasets: Big data sets available for free

Improve this answer
$\endgroup$
16
$\begingroup$

Did you know about the PUMA Benchmarks and dataset downloads? https://sites.google.com/site/farazahmad/pumadatasets

It does include the following:

  1. TeraSort
  2. Wikipedia
  3. List item
  4. Self-Join
  5. Adjacency-List
  6. Movies-database
  7. Ranked-Inverted-Index
Improve this answer
$\endgroup$
16
$\begingroup$

The UK Government provide an excellent source of non-personal data collected throughout government departments: http://data.gov.uk

Improve this answer
$\endgroup$
16
$\begingroup$

Data Sets

Data Sets From awesome-datascience

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Can you please provide us with some information on both datasets/links? This will indeed ease the burden of those looking for specific types of data set. Take a look at other posts to see what kind of information your references are missing. $\endgroup$
    – Rubens
    Commented Jan 30, 2015 at 21:31
15
$\begingroup$

I am new to this forum. Chiming in late on this question. I have been maintaining (I am a co-founder of) a catalog of publicly available data portals. There is over 1000 now listed and cover portals at international, federal, state, municipal and academic levels across the globe.

http://www.opengeocode.org/opendata/

Improve this answer
$\endgroup$
15
$\begingroup$

I'm surprised one has not mentioned this, as it seems fairly obvious: http://www.kaggle.com consistently has new and very interesting datasets. Information is considered an asset, so often companies don't want to release that data (plus privacy concerns). Kaggle gives you data and they hope you solve business problems with it in exchange.

Improve this answer
$\endgroup$
12
$\begingroup$

As you mentioned, the API is the hard part, not the data. Quandl seems to solve this problem by providing over 10 million publicly available data sets under one easy, RESTful API. If programming isn't your strong suit, there is a free tool to make loading data into Excel very easy. Additionally, if you do enjoy programming, there are several native libraries in R, Python, Java and more.

Improve this answer
$\endgroup$
12
$\begingroup$

To add to a possibly never ending list:

as mentioned by cyndd, there is Wikidata,

and for curated structured knowledge, Wolfram Alpha.

Improve this answer
$\endgroup$
12
$\begingroup$

I came across this collection on Github. The collection is categorised as well.

https://github.com/caesar0301/awesome-public-datasets

And for the part regarding

Can not a open-source model for sharing data sets devised for data scientists?

you can refer The Leek group guide to data sharing

Improve this answer
$\endgroup$
10
$\begingroup$

Not all government data is listed on data.gov - Sunlight Foundation put together a set of spreadsheets back in February describing sets of available data.

Improve this answer
$\endgroup$
9
$\begingroup$

One other data source I didn't see listed is The GDELT Project. From the site:

GDELT Project monitors the world's broadcast, print, and web news from nearly every corner of every country in over 100 languages and identifies the people, locations, organizations, counts, themes, sources, and events driving our global society every second of every day, creating a free open platform for computing on the entire world.

Improve this answer
$\endgroup$
8
$\begingroup$

This subreddit lists a lot of known Datasets

Reddit Datasets

There are a lot of dataset requests on that subreddit, several of which have been answered.

Improve this answer
$\endgroup$
6
$\begingroup$

I created a github repo for this. The datasets are not big, but are minimal examples meant to practice and explore predictive-modeling techniques which can then be extended to big datasets.

Machine Learning Problem Bible (MLPB)

The cool/unique thing about this repo is that every problem is tagged with tags like [multi-class], [unbalanced-data], [regression], etc. making it easy to find certain types of problems/datasets.

Improve this answer
$\endgroup$
6
$\begingroup$

Eurostats http://ec.europa.eu/eurostat and European Central Bank https://www.ecb.europa.eu/stats/html/index.en.html provide a great variety of datasets which I use quite often in my work projects.

Improve this answer
$\endgroup$
6
$\begingroup$

Besides all these datasets, if you are interested in data related to India. The publicly official site of Indian Government is

It provides datasets from different departments of Indian government which can be well used for Big Data Analysis & Machine Learning.

Improve this answer
$\endgroup$
4
$\begingroup$

Yahoo just released a huge dataset for research community. Enjoy it!

Improve this answer
$\endgroup$
4
$\begingroup$

Just we load MASS package in R we access multiple dataframes or data sets .

install.packages("MASS") require("MASS")

Improve this answer
$\endgroup$
3
$\begingroup$

3 datasets from https://www.jc-bingo.com/about

  • visitor-interests.csv Aggregated visitor interests compiled based on 1 week web access logs. Includes visitor IP address, user-agent string, visitor country, accessed page languages and topics. 19,926 records, 2.9 Mb.
  • user-agents.csv Real visitor user agents ordered by popularity. 4,826 records, 716 Kb.
  • bots.csv Robot IP addresses and user-agent strings extracted from web access logs. 1,293 records, 122 Kb.
Improve this answer
$\endgroup$
3
$\begingroup$

Obviously, there exists a large set of public databases.

One not yet mentioned, is from the FAO (Food and Agriculture Organization of the United Nations), accessible at:

http://www.fao.org/faostat/

It contains data about food production for worldwide countries.

Improve this answer
$\endgroup$
1
2

Not the answer you're looking for? Browse other questions tagged or ask your own question.