Questions tagged [dataset]
A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData
1,480
questions
202
votes
35
answers
33k
views
Publicly Available Datasets
One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...
59
votes
6
answers
16k
views
Should I go for a 'balanced' dataset or a 'representative' dataset?
My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
- 693
55
votes
5
answers
29k
views
Is it always better to use the whole dataset to train the final model?
A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, ...
- 3,920
35
votes
9
answers
13k
views
Why is it wrong to train and test a model on the same dataset?
What are the pitfalls of doing so and why is it a bad practice? Is it possible that the model starts to learn the images "by heart" instead of understanding the underlying logic?
- 461
35
votes
4
answers
16k
views
Quick guide into training highly imbalanced data sets
I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
- 5,444
28
votes
7
answers
27k
views
Publicly available social network datasets/APIs
As an extension to our great list of publicly available datasets, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside ...
- 4,097
28
votes
3
answers
43k
views
Data Science Project Ideas [closed]
I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion.
I have just started with Data Science and ...
- 383
25
votes
4
answers
16k
views
Is there any data tidying tool for python/pandas similar to R tidyr tool?
I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of ...
- 807
23
votes
6
answers
128k
views
Uploading images folder from my system into Google Colab
I want to train a deep learning model on a dataset containing around 3000 images. Since the dataset is huge, I want to use Google colab since it's GPU supported. How do I upload this full image folder ...
- 341
22
votes
3
answers
14k
views
How to generate synthetic dataset using machine learning model learnt with original dataset?
Generally, the machine learning model is built on datasets. I'd like to know if there is any way to generate synthetic dataset using such trained machine learning model preserving original dataset ...
- 323
22
votes
2
answers
44k
views
Loading own train data and labels in dataloader using pytorch?
I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader?
I have a dataset that I created and the ...
- 351
21
votes
3
answers
10k
views
Dataset for Named Entity Recognition on Informal Text
I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the ...
- 2,029
18
votes
4
answers
28k
views
One hot encoding alternatives for large categorical values
I have a data frame with large categorical values over 1600 categories. Is there any way I can find alternatives so that I don't have over 1600 columns?
I found this interesting link.
But they are ...
- 283
17
votes
3
answers
9k
views
Why are variables of train and test data defined using the capital letter (in Python)?
I hope this question is the most suitable in this site...
In Python, usually the class name is defined using the capital letter as its first character, for example
...
- 901
17
votes
2
answers
95k
views
How to store strings in CSV with new line characters?
My question is: what are ways I can store strings in a CSV that contain newline characters (i.e. \n), where each data point is in one line?
Sample data
This is a ...
- 3,498
17
votes
3
answers
5k
views
With unbalanced class, do I have to use under sampling on my validation/testing datasets?
I’m a beginner in machine learning and I’m facing a situation. I’m working on a Real Time Bidding problem, with the IPinYou dataset and I’m trying to do a click prediction.
The thing is that, as you ...
- 619
17
votes
3
answers
22k
views
When should we consider a dataset as imbalanced?
I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced.
My question is, are there any rules of thumb that tell us when we should subsample the large ...
- 594
16
votes
5
answers
24k
views
Downloading a large dataset on the web directly into AWS S3
Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL?
Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I ...
- 163
16
votes
2
answers
14k
views
How much data are sufficient to train my machine learning model?
I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining.
My colleague (who is a machine ...
- 281
15
votes
3
answers
16k
views
Is there a person class in ImageNet? Are there any classes related to humans?
If I look at one of the many sources for the Imagenet classes on the Internet I cannot find a single class related to human beings (and no, harvestman is not someone who harvests, but it's what I knew ...
- 399
15
votes
2
answers
4k
views
Why do we need to handle data imbalance?
I would like to know why we need to deal with data imbalance. I know how to deal with it and different methods to solve the issue - by up sampling or down sampling or by using SMOTE.
For example, if I ...
- 481
15
votes
2
answers
5k
views
Analyzing A/B test results which are not normally distributed, using independent t-test
I have a set of results from an A/B test (one control group, one feature group) which do not fit a Normal Distribution.
In fact the distribution resembles more closely the Landau Distribution.
I ...
- 253
14
votes
3
answers
30k
views
Where can I download historical market capitalization and daily turnover data for stocks?
There are plenty of sources which provide the historical stock data but they only provide the OHLC fields along with volume and adjusted close. Also a couple of sources I found provide market cap data ...
- 3,985
13
votes
6
answers
815
views
Datasets understanding best practices
I am a CS master student in data mining. My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean ...
- 719
13
votes
1
answer
4k
views
Do I have to standardize my new polynomial features?
I have a vector X with n features previously standardized.
If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
- 619
12
votes
3
answers
2k
views
Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?
I want to investigate price-setting behavior of airlines -- specifically how airlines react to competitors pricing.
As I would say my knowledge about more complex analysis is quite limited I've done ...
- 221
12
votes
1
answer
4k
views
Can HDF5 be reliably written to and read from simultaneously by separate python processes?
I'm writing a script to record live data over time into a single HDF5 file which includes my whole dataset for this project. I'm working with Python 3.6 and decided to create a command line tool using ...
- 297
11
votes
6
answers
28k
views
Python: Handling imbalance Classes in python Machine Learning
I have a dataset for which I am trying to predict target variables.
...
- 1,045
11
votes
5
answers
5k
views
Covariate shift detection
Is there any standard approach for detecting the covariate shift between the training and test data ? This would be useful to validate the assumption that covariate shift exists in my database which ...
- 181
11
votes
2
answers
12k
views
How to split train/test in recommender systems
I am working with the MovieLens10M dataset, predicting user ratings. If I want to fairly evaluate my algorithm, how should I split my training v. test data?
By default, I believe the data is split ...
- 3,077
10
votes
3
answers
20k
views
Public dataset for news articles with their associated categories
I am wondering if there are any public datasets of Google news with various news categories such as politics, entertainment, lifestyle, general news, sports etc.
I want to use such dataset for topic ...
- 213
10
votes
2
answers
5k
views
how to check the distribution of the training set and testing set are similar
I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of ...
- 277
10
votes
2
answers
7k
views
Benchmark datasets for collaborative filtering
I'd like to test a new algorithm for collaborative filtering. A typical use case is to recommend movies based on the preferences of users similar to the specific user.
What are some common benchmark ...
- 790
10
votes
3
answers
1k
views
NASDAQ Trade Data
I am trying to find stock data to practice with, is there a good resource for this? I found this but it only has the current year.
I already have a way of parsing the protocol, but would like to have ...
- 103
10
votes
3
answers
948
views
Network analysis classic datasets
There are several classic datasets for machine learning classification/regression tasks. The most popular are:
Iris Flower Data Set;
Titanic Data Set;
Motor Trend Cars;
etc.
But does anyone know ...
- 1,119
10
votes
3
answers
8k
views
Splitting train/test sets by an identifier?
I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, ...
- 323
10
votes
1
answer
9k
views
How do you calculate how dense or sparse a dataset is?
I'm looking deeper into collaborative filtering. One really interesting paper is "A Comparative Study of Collaborative Filtering Algorithms" http://arxiv.org/pdf/1205.3193.pdf
In order to select ...
- 203
10
votes
6
answers
1k
views
What are some of the best practices for sharing data and models with colleagues?
As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
9
votes
4
answers
3k
views
Suggest text classifier training datasets
Which freely available datasets can I use to train a text classifier?
We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified ...
- 121
9
votes
1
answer
24k
views
How can I get the ImageNet ILSVRC 2012 data used for the classification challenge?
I would like to see if I can reproduce some of the image net results. However, I could not find the data (the list of URLs) used for training / testing in the ILSVRC 2012 (or later) classification ...
- 18.7k
9
votes
3
answers
387
views
Is (nearly) all data separable?
Suppose I have some data set with two classes. I could draw a decision boundary around each data point belonging to one of these classes, and hence, separate the data, like so:
Where the red lines ...
- 467
9
votes
3
answers
1k
views
Interactive Graphing while logging data
I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to ...
9
votes
2
answers
24k
views
Always drop the first column after performing One Hot Encoding?
Since one of the columns can be generated completely from the others, and hence retaining this extra column does not add any new information for the modelling process, would it be good practice to ...
- 403
9
votes
2
answers
967
views
How to model user's buying behavior on Amazon?
For our final course project in Data Science, we proposed the following-
Give the Amazon Reviews Dataset, we plan to come up with an algorithm (thats roughly based on Personalized PageRank) that ...
- 191
8
votes
4
answers
10k
views
One Hot encoding for large number of values
How do we use one hot encoding if the number of values which a categorical variable can take is large ?
In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary ...
- 367
8
votes
3
answers
3k
views
What is normalization for?
I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do?
...
- 207
8
votes
1
answer
8k
views
What is the distribution of categories in imagenet training set (ILSVRC2012)
http://arxiv.org/pdf/1409.0575v3.pdf Table 2
says there are 1,281,167 images and 732-1300 per class in the ILSVRC2012 training set.
Ideally I'd like to avoid downloading the 138 GB just for this ...
- 81
8
votes
1
answer
14k
views
Is it possible to have stratified train-test split of a set based on two columns?
Consider a dataframe that contains two columns, text and label. I can very easily create a stratified train-test split using ...
- 203
8
votes
4
answers
2k
views
Job title similarity
I'm trying to define a metric between job titles in IT field. For this I need some metric between words of job titles that are not appearing together in the same job title, e.g. metric between the ...
- 181
8
votes
3
answers
8k
views
Where can I find freely available multi-label datasets online? [closed]
I'm trying to find multi-label classfication datasets, which are available for free online.
By "multi-label" I mean that each instance can be labeled with anywhere from a single to $k$ labels, where ...
- 185