Questions tagged [dataset]
A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData
926
questions
173
votes
35answers
28k views
Publicly Available Datasets
One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...
49
votes
5answers
11k views
Should I go for a 'balanced' dataset or a 'representative' dataset?
My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
29
votes
4answers
12k views
Quick guide into training highly imbalanced data sets
I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
27
votes
4answers
10k views
Is it always better to use the whole dataset to train the final model?
A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, ...
26
votes
7answers
25k views
Publicly available social network datasets/APIs
As an extension to our great list of publicly available datasets, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside ...
24
votes
3answers
42k views
Data Science Project Ideas [closed]
I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion.
I have just started with Data Science and ...
19
votes
3answers
13k views
How to generate synthetic dataset using machine learning model learnt with original dataset?
Generally, the machine learning model is built on datasets. I'd like to know if there is any way to generate synthetic dataset using such trained machine learning model preserving original dataset ...
19
votes
3answers
12k views
is there any data tidying tool for python/pandas similar to R tidyr tool?
I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of gather(), separate() and ...
18
votes
3answers
8k views
Dataset for Named Entity Recognition on Informal Text
I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the ...
14
votes
3answers
7k views
Is there a person class in ImageNet? Are there any classes related to humans?
If I look at one of the many sources for the Imagenet classes on the Internet I cannot find a single class related to human beings (and no, harvestman is not someone who harvests, but it's what I knew ...
13
votes
2answers
5k views
Why are variables of train and test data defined using the capital letter (in Python)?
I hope this question is the most suitable in this site...
In Python, usually the class name is defined using the capital letter as its first character, for example
...
13
votes
6answers
481 views
Datasets understanding best practices
I am a CS master student in data mining. My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean ...
13
votes
3answers
4k views
With unbalanced class, do I have to use under sampling on my validation/testing datasets?
I’m a beginner in machine learning and I’m facing a situation. I’m working on a Real Time Bidding problem, with the IPinYou dataset and I’m trying to do a click prediction.
The thing is that, as you ...
13
votes
2answers
2k views
Analyzing A/B test results which are not normally distributed, using independent t-test
I have a set of results from an A/B test (one control group, one feature group) which do not fit a Normal Distribution.
In fact the distribution resembles more closely the Landau Distribution.
I ...
12
votes
4answers
12k views
Downloading a large dataset on the web directly into AWS S3
Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL?
Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I ...
12
votes
2answers
1k views
Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?
I want to investigate price-setting behavior of airlines -- specifically how airlines react to competitors pricing.
As I would say my knowledge about more complex analysis is quite limited I've done ...
12
votes
2answers
2k views
why we need to handle data imbalance?
I need to know why we need to deal with data imbalance. I know how to deal with it and different methods to solve the issue which is by up sampling or down sampling or by using Smote.
For example, if ...
11
votes
4answers
13k views
One hot encoding alternatives for large categorical values?
Hi have dataframe with large categorical values over 1600 categories is there any way I can find alternatives so that I don't have over 1600 columns.
I found this below interesting link http://...
11
votes
4answers
15k views
Where can I download historical market capitalization and daily turnover data for stocks?
There are plenty of sources which provide the historical stock data but they only provide the OHLC fields along with volume and adjusted close. Also a couple of sources I found provide market cap data ...
11
votes
2answers
8k views
How much data are sufficient to train my machine learning model?
I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining.
My colleague (who is a machine ...
10
votes
5answers
48k views
Uploading images folder from my system into Google Colab
I want to train a deep learning model on a dataset containing around 3000 images. Since the dataset is huge, I want to use Google colab since it's GPU supported. How do I upload this full image folder ...
10
votes
3answers
12k views
When should we consider a dataset as imbalanced?
I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced.
My question is, are there any rules of thumb that tell us when we should subsample the large ...
10
votes
3answers
1k views
NASDAQ Trade Data
I am trying to find stock data to practice with, is there a good resource for this? I found this: ftp://emi.nasdaq.com/ITCH/ but it only has the current year.
I already have a way of parsing the ...
9
votes
4answers
3k views
Suggest text classifier training datasets
Which freely available datasets can I use to train a text classifier?
We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified ...
9
votes
6answers
22k views
Python: Handling imbalance Classes in python Machine Learning
I have a dataset for which I am trying to predict target variables.
...
9
votes
1answer
2k views
Can HDF5 be reliably written to and read from simultaneously by separate python processes?
I'm writing a script to record live data over time into a single HDF5 file which includes my whole dataset for this project. I'm working with Python 3.6 and decided to create a command line tool using ...
9
votes
2answers
4k views
Benchmark datasets for collaborative filtering
I'd like to test a new algorithm for collaborative filtering. A typical use case is to recommend movies based on the preferences of users similar to the specific user.
What are some common benchmark ...
9
votes
3answers
659 views
Network analysis classic datasets
There are several classic datasets for machine learning classification/regression tasks. The most popular are:
Iris Flower Data Set;
Titanic Data Set;
Motor Trend Cars;
etc.
But does anyone know ...
9
votes
2answers
871 views
How to model user's buying behavior on Amazon?
For our final course project in Data Science, we proposed the following-
Give the Amazon Reviews Dataset, we plan to come up with an algorithm (thats roughly based on Personalized PageRank) that ...
8
votes
3answers
950 views
Interactive Graphing while logging data
I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to ...
8
votes
4answers
1k views
Job title similarity
I'm trying to define a metric between job titles in IT field. For this I need some metric between words of job titles that are not appearing together in the same job title, e.g. metric between the ...
8
votes
2answers
750 views
Best way to store large data set using R from Twitter?
I am working on a project that aims to retrieve a large data-set (i.e., tweet data which is a couple of days old) from Twitter using the twitteR library on R. have difficulty storing tweets because ...
8
votes
3answers
108 views
Is (nearly) all data separable?
Suppose I have some data set with two classes. I could draw a decision boundary around each data point belonging to one of these classes, and hence, separate the data, like so:
Where the red lines ...
8
votes
1answer
6k views
How to split train/test in recommender systems
I am working with the MovieLens10M dataset, predicting user ratings. If I want to fairly evaluate my algorithm, how should I split my training v. test data?
By default, I believe the data is split ...
8
votes
5answers
470 views
What are some of the best practices for sharing data and models with colleagues?
As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
7
votes
5answers
4k views
Where can I find free spatio-temporal dataset for download?
Where can I find free spatio-temporal dataset for download so that I can play with it in R ?
7
votes
1answer
20k views
How can I get the ImageNet ILSVRC 2012 data used for the classification challenge?
I would like to see if I can reproduce some of the image net results. However, I could not find the data (the list of URLs) used for training / testing in the ILSVRC 2012 (or later) classification ...
7
votes
3answers
5k views
One Hot encoding for large number of values
How do we use one hot encoding if the number of values which a categorical variable can take is large ?
In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary ...
7
votes
2answers
704 views
how to check the distribution of the training set and testing set are similar
I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of ...
7
votes
1answer
6k views
How do you calculate how dense or sparse a dataset is?
I'm looking deeper into collaborative filtering. One really interesting paper is "A Comparative Study of Collaborative Filtering Algorithms" http://arxiv.org/pdf/1205.3193.pdf
In order to select ...
7
votes
5answers
3k views
Covariate shift detection
Is there any standard approach for detecting the covariate shift between the training and test data ? This would be useful to validate the assumption that covariate shift exists in my database which ...
7
votes
2answers
827 views
Matrix properties and machine learning/data mining
I'm doing some data analysis in a Statistical Pattern Recognition course using PRML. We analyzed a lot of matrix properties, like eigenvalues, column independence, positive semi-definite matrix, etc. ...
7
votes
2answers
3k views
Where can I find freely available multi-label datasets online?
I'm trying to find multi-label classfication datasets, which are available for free online.
By "multi-label" I mean that each instance can be labeled with anywhere from a single to $k$ labels, where ...
7
votes
1answer
192 views
Evaluating Recommendation engines
What is the standard way for evaluating and comparing different algorithms while developing recommendation system? Whether we need to have a predetermined annotated ranked dataset and then compare ...
7
votes
1answer
205 views
On a multi lingual sentiment corpus
I am looking to compile a sentiment corpus for news articles in multiple languages (~100k per lang. for a machine learning experiment) where each article is labeled positive, neutral, or negative. I ...
7
votes
2answers
954 views
Difference between training and test data distributions
A basic assumption in machine learning is that training and test data are drawn from the same population, and thus follow the same distribution. But, in practice, this is highly unlikely. Covariate ...
6
votes
2answers
5k views
For which real world data sets does DBSCAN surpass K-means.?
For clustering, DBSCAN surpass k-means in terms of handling arbitrary shape data sets. In the most published papers about density based clustering, the experiments are performed with synthetic data ...
6
votes
3answers
3k views
What is normalization for?
I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do?
...
6
votes
3answers
11k views
Public dataset for news articles with their associated categories
I am wondering if there are any public datasets of Google news with various news categories such as politics, entertainment, lifestyle, general news, sports etc.
I want to use such dataset for topic ...
6
votes
2answers
52 views
When to use missing data imputation in the data analysis problem?
I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
