Questions tagged [dataset]
A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData
1,137
questions
190
votes
35answers
30k views
Publicly Available Datasets
One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...
52
votes
6answers
13k views
Should I go for a 'balanced' dataset or a 'representative' dataset?
My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
35
votes
4answers
14k views
Is it always better to use the whole dataset to train the final model?
A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, ...
34
votes
4answers
14k views
Quick guide into training highly imbalanced data sets
I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
28
votes
8answers
5k views
Why is it wrong to train and test a model on the same dataset?
What are the pitfalls of doing so and why is it a bad practice? Is it possible that the model starts to learn the images by heart instead of understanding the underlying logic?
26
votes
7answers
26k views
Publicly available social network datasets/APIs
As an extension to our great list of publicly available datasets, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside ...
25
votes
3answers
43k views
Data Science Project Ideas [closed]
I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion.
I have just started with Data Science and ...
22
votes
3answers
15k views
Is there any data tidying tool for python/pandas similar to R tidyr tool?
I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of ...
20
votes
3answers
13k views
How to generate synthetic dataset using machine learning model learnt with original dataset?
Generally, the machine learning model is built on datasets. I'd like to know if there is any way to generate synthetic dataset using such trained machine learning model preserving original dataset ...
20
votes
6answers
89k views
Uploading images folder from my system into Google Colab
I want to train a deep learning model on a dataset containing around 3000 images. Since the dataset is huge, I want to use Google colab since it's GPU supported. How do I upload this full image folder ...
19
votes
3answers
9k views
Dataset for Named Entity Recognition on Informal Text
I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the ...
15
votes
2answers
6k views
Why are variables of train and test data defined using the capital letter (in Python)?
I hope this question is the most suitable in this site...
In Python, usually the class name is defined using the capital letter as its first character, for example
...
15
votes
3answers
5k views
With unbalanced class, do I have to use under sampling on my validation/testing datasets?
I’m a beginner in machine learning and I’m facing a situation. I’m working on a Real Time Bidding problem, with the IPinYou dataset and I’m trying to do a click prediction.
The thing is that, as you ...
15
votes
2answers
11k views
How much data are sufficient to train my machine learning model?
I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining.
My colleague (who is a machine ...
14
votes
3answers
10k views
Is there a person class in ImageNet? Are there any classes related to humans?
If I look at one of the many sources for the Imagenet classes on the Internet I cannot find a single class related to human beings (and no, harvestman is not someone who harvests, but it's what I knew ...
14
votes
4answers
20k views
One hot encoding alternatives for large categorical values
I have a data frame with large categorical values over 1600 categories. Is there any way I can find alternatives so that I don't have over 1600 columns?
I found this interesting link.
But they are ...
14
votes
2answers
3k views
why we need to handle data imbalance?
I need to know why we need to deal with data imbalance. I know how to deal with it and different methods to solve the issue which is by up sampling or down sampling or by using Smote.
For example, if ...
14
votes
2answers
3k views
Analyzing A/B test results which are not normally distributed, using independent t-test
I have a set of results from an A/B test (one control group, one feature group) which do not fit a Normal Distribution.
In fact the distribution resembles more closely the Landau Distribution.
I ...
13
votes
6answers
623 views
Datasets understanding best practices
I am a CS master student in data mining. My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean ...
12
votes
5answers
17k views
Downloading a large dataset on the web directly into AWS S3
Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL?
Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I ...
12
votes
3answers
16k views
When should we consider a dataset as imbalanced?
I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced.
My question is, are there any rules of thumb that tell us when we should subsample the large ...
12
votes
2answers
19k views
Loading own train data and labels in dataloader using pytorch?
I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader?
I have a dataset that I created and the ...
12
votes
2answers
1k views
Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?
I want to investigate price-setting behavior of airlines -- specifically how airlines react to competitors pricing.
As I would say my knowledge about more complex analysis is quite limited I've done ...
12
votes
4answers
22k views
Where can I download historical market capitalization and daily turnover data for stocks?
There are plenty of sources which provide the historical stock data but they only provide the OHLC fields along with volume and adjusted close. Also a couple of sources I found provide market cap data ...
10
votes
1answer
3k views
Can HDF5 be reliably written to and read from simultaneously by separate python processes?
I'm writing a script to record live data over time into a single HDF5 file which includes my whole dataset for this project. I'm working with Python 3.6 and decided to create a command line tool using ...
10
votes
6answers
25k views
Python: Handling imbalance Classes in python Machine Learning
I have a dataset for which I am trying to predict target variables.
...
10
votes
2answers
41k views
How to store strings in CSV with new line characters?
My question is: what are ways I can store strings in a CSV that contain newline characters (i.e. \n), where each data point is in one line?
Sample data
This is a ...
10
votes
3answers
1k views
NASDAQ Trade Data
I am trying to find stock data to practice with, is there a good resource for this? I found this but it only has the current year.
I already have a way of parsing the protocol, but would like to have ...
10
votes
3answers
724 views
Network analysis classic datasets
There are several classic datasets for machine learning classification/regression tasks. The most popular are:
Iris Flower Data Set;
Titanic Data Set;
Motor Trend Cars;
etc.
But does anyone know ...
10
votes
2answers
5k views
Benchmark datasets for collaborative filtering
I'd like to test a new algorithm for collaborative filtering. A typical use case is to recommend movies based on the preferences of users similar to the specific user.
What are some common benchmark ...
9
votes
4answers
3k views
Suggest text classifier training datasets
Which freely available datasets can I use to train a text classifier?
We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified ...
9
votes
1answer
22k views
How can I get the ImageNet ILSVRC 2012 data used for the classification challenge?
I would like to see if I can reproduce some of the image net results. However, I could not find the data (the list of URLs) used for training / testing in the ILSVRC 2012 (or later) classification ...
9
votes
3answers
16k views
Public dataset for news articles with their associated categories
I am wondering if there are any public datasets of Google news with various news categories such as politics, entertainment, lifestyle, general news, sports etc.
I want to use such dataset for topic ...
9
votes
2answers
2k views
how to check the distribution of the training set and testing set are similar
I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of ...
9
votes
3answers
178 views
Is (nearly) all data separable?
Suppose I have some data set with two classes. I could draw a decision boundary around each data point belonging to one of these classes, and hence, separate the data, like so:
Where the red lines ...
9
votes
5answers
4k views
Covariate shift detection
Is there any standard approach for detecting the covariate shift between the training and test data ? This would be useful to validate the assumption that covariate shift exists in my database which ...
9
votes
3answers
974 views
Interactive Graphing while logging data
I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to ...
9
votes
1answer
7k views
How do you calculate how dense or sparse a dataset is?
I'm looking deeper into collaborative filtering. One really interesting paper is "A Comparative Study of Collaborative Filtering Algorithms" http://arxiv.org/pdf/1205.3193.pdf
In order to select ...
9
votes
1answer
8k views
How to split train/test in recommender systems
I am working with the MovieLens10M dataset, predicting user ratings. If I want to fairly evaluate my algorithm, how should I split my training v. test data?
By default, I believe the data is split ...
9
votes
6answers
729 views
What are some of the best practices for sharing data and models with colleagues?
As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
9
votes
2answers
919 views
How to model user's buying behavior on Amazon?
For our final course project in Data Science, we proposed the following-
Give the Amazon Reviews Dataset, we plan to come up with an algorithm (thats roughly based on Personalized PageRank) that ...
8
votes
3answers
3k views
What is normalization for?
I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do?
...
8
votes
3answers
6k views
One Hot encoding for large number of values
How do we use one hot encoding if the number of values which a categorical variable can take is large ?
In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary ...
8
votes
2answers
4k views
Splitting train/test sets by an identifier?
I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, ...
8
votes
4answers
1k views
Job title similarity
I'm trying to define a metric between job titles in IT field. For this I need some metric between words of job titles that are not appearing together in the same job title, e.g. metric between the ...
8
votes
2answers
795 views
Best way to store large data set using R from Twitter?
I am working on a project that aims to retrieve a large data-set (i.e., tweet data which is a couple of days old) from Twitter using the twitteR library on R. have difficulty storing tweets because ...
8
votes
3answers
5k views
Where can I find freely available multi-label datasets online? [closed]
I'm trying to find multi-label classfication datasets, which are available for free online.
By "multi-label" I mean that each instance can be labeled with anywhere from a single to $k$ labels, where ...
8
votes
2answers
894 views
Matrix properties and machine learning/data mining
I'm doing some data analysis in a Statistical Pattern Recognition course using PRML. We analyzed a lot of matrix properties, like eigenvalues, column independence, positive semi-definite matrix, etc. ...
8
votes
1answer
2k views
Do I have to standardize my new polynomial features?
I have a vector X with n features previously standardized.
If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
8
votes
1answer
211 views
Evaluating Recommendation engines
What is the standard way for evaluating and comparing different algorithms while developing recommendation system? Whether we need to have a predetermined annotated ranked dataset and then compare ...
