Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
202 votes
35 answers
33k views

Publicly Available Datasets

One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...
59 votes
6 answers
16k views

Should I go for a 'balanced' dataset or a 'representative' dataset?

My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
pnp's user avatar
  • 693
55 votes
5 answers
29k views

Is it always better to use the whole dataset to train the final model?

A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, ...
pcko1's user avatar
  • 3,920
35 votes
9 answers
13k views

Why is it wrong to train and test a model on the same dataset?

What are the pitfalls of doing so and why is it a bad practice? Is it possible that the model starts to learn the images "by heart" instead of understanding the underlying logic?
karalis1's user avatar
  • 461
35 votes
4 answers
16k views

Quick guide into training highly imbalanced data sets

I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
IgorS's user avatar
  • 5,444
28 votes
7 answers
27k views

Publicly available social network datasets/APIs

As an extension to our great list of publicly available datasets, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside ...
Rubens's user avatar
  • 4,097
28 votes
3 answers
43k views

Data Science Project Ideas [closed]

I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion. I have just started with Data Science and ...
Kevin Desai's user avatar
25 votes
4 answers
16k views

Is there any data tidying tool for python/pandas similar to R tidyr tool?

I'm working on a Kaggle challenge where some variables are represented by rows instead of columns (Telstra Network Disruption). I am currently searching for the equivalent of ...
cpumar's user avatar
  • 807
23 votes
6 answers
128k views

Uploading images folder from my system into Google Colab

I want to train a deep learning model on a dataset containing around 3000 images. Since the dataset is huge, I want to use Google colab since it's GPU supported. How do I upload this full image folder ...
chatbot_chakra's user avatar
22 votes
3 answers
14k views

How to generate synthetic dataset using machine learning model learnt with original dataset?

Generally, the machine learning model is built on datasets. I'd like to know if there is any way to generate synthetic dataset using such trained machine learning model preserving original dataset ...
m-bhole's user avatar
  • 323
22 votes
2 answers
44k views

Loading own train data and labels in dataloader using pytorch?

I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader? I have a dataset that I created and the ...
Amarnath's user avatar
  • 351
21 votes
3 answers
10k views

Dataset for Named Entity Recognition on Informal Text

I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the ...
Madison May's user avatar
  • 2,029
18 votes
4 answers
28k views

One hot encoding alternatives for large categorical values

I have a data frame with large categorical values over 1600 categories. Is there any way I can find alternatives so that I don't have over 1600 columns? I found this interesting link. But they are ...
vinaykva's user avatar
  • 283
17 votes
3 answers
9k views

Why are variables of train and test data defined using the capital letter (in Python)?

I hope this question is the most suitable in this site... In Python, usually the class name is defined using the capital letter as its first character, for example ...
Blaszard's user avatar
  • 901
17 votes
2 answers
95k views

How to store strings in CSV with new line characters?

My question is: what are ways I can store strings in a CSV that contain newline characters (i.e. \n), where each data point is in one line? Sample data This is a ...
Bruno Lubascher's user avatar
17 votes
3 answers
5k views

With unbalanced class, do I have to use under sampling on my validation/testing datasets?

I’m a beginner in machine learning and I’m facing a situation. I’m working on a Real Time Bidding problem, with the IPinYou dataset and I’m trying to do a click prediction. The thing is that, as you ...
jmvllt's user avatar
  • 619
17 votes
3 answers
22k views

When should we consider a dataset as imbalanced?

I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced. My question is, are there any rules of thumb that tell us when we should subsample the large ...
Rami's user avatar
  • 594
16 votes
5 answers
24k views

Downloading a large dataset on the web directly into AWS S3

Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL? Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I ...
Will Stedden's user avatar
16 votes
2 answers
14k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
DavideChicco.it's user avatar
15 votes
3 answers
16k views

Is there a person class in ImageNet? Are there any classes related to humans?

If I look at one of the many sources for the Imagenet classes on the Internet I cannot find a single class related to human beings (and no, harvestman is not someone who harvests, but it's what I knew ...
DeltaIV's user avatar
  • 399
15 votes
2 answers
4k views

Why do we need to handle data imbalance?

I would like to know why we need to deal with data imbalance. I know how to deal with it and different methods to solve the issue - by up sampling or down sampling or by using SMOTE. For example, if I ...
sara's user avatar
  • 481
15 votes
2 answers
5k views

Analyzing A/B test results which are not normally distributed, using independent t-test

I have a set of results from an A/B test (one control group, one feature group) which do not fit a Normal Distribution. In fact the distribution resembles more closely the Landau Distribution. I ...
teebszet's user avatar
  • 253
14 votes
3 answers
30k views

Where can I download historical market capitalization and daily turnover data for stocks?

There are plenty of sources which provide the historical stock data but they only provide the OHLC fields along with volume and adjusted close. Also a couple of sources I found provide market cap data ...
tejaskhot's user avatar
  • 3,985
13 votes
6 answers
815 views

Datasets understanding best practices

I am a CS master student in data mining. My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean ...
Jack Twain's user avatar
13 votes
1 answer
4k views

Do I have to standardize my new polynomial features?

I have a vector X with n features previously standardized. If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
jmvllt's user avatar
  • 619
12 votes
3 answers
2k views

Airline Fares - What analysis should be used to detect competitive price-setting behavior and price correlations?

I want to investigate price-setting behavior of airlines -- specifically how airlines react to competitors pricing. As I would say my knowledge about more complex analysis is quite limited I've done ...
s1x's user avatar
  • 221
12 votes
1 answer
4k views

Can HDF5 be reliably written to and read from simultaneously by separate python processes?

I'm writing a script to record live data over time into a single HDF5 file which includes my whole dataset for this project. I'm working with Python 3.6 and decided to create a command line tool using ...
basse's user avatar
  • 297
11 votes
6 answers
28k views

Python: Handling imbalance Classes in python Machine Learning

I have a dataset for which I am trying to predict target variables. ...
SRS's user avatar
  • 1,045
11 votes
5 answers
5k views

Covariate shift detection

Is there any standard approach for detecting the covariate shift between the training and test data ? This would be useful to validate the assumption that covariate shift exists in my database which ...
Daniel Wonglee's user avatar
11 votes
2 answers
12k views

How to split train/test in recommender systems

I am working with the MovieLens10M dataset, predicting user ratings. If I want to fairly evaluate my algorithm, how should I split my training v. test data? By default, I believe the data is split ...
jamesmf's user avatar
  • 3,077
10 votes
3 answers
20k views

Public dataset for news articles with their associated categories

I am wondering if there are any public datasets of Google news with various news categories such as politics, entertainment, lifestyle, general news, sports etc. I want to use such dataset for topic ...
utengr's user avatar
  • 213
10 votes
2 answers
5k views

how to check the distribution of the training set and testing set are similar

I have been playing the Kaggle Competition and I find there is a situation that the distribution of the training set and testing set are different, so I am wondering how to check the distribution of ...
Bowen Peng's user avatar
10 votes
2 answers
7k views

Benchmark datasets for collaborative filtering

I'd like to test a new algorithm for collaborative filtering. A typical use case is to recommend movies based on the preferences of users similar to the specific user. What are some common benchmark ...
pir's user avatar
  • 790
10 votes
3 answers
1k views

NASDAQ Trade Data

I am trying to find stock data to practice with, is there a good resource for this? I found this but it only has the current year. I already have a way of parsing the protocol, but would like to have ...
Marin's user avatar
  • 103
10 votes
3 answers
948 views

Network analysis classic datasets

There are several classic datasets for machine learning classification/regression tasks. The most popular are: Iris Flower Data Set; Titanic Data Set; Motor Trend Cars; etc. But does anyone know ...
sobach's user avatar
  • 1,119
10 votes
3 answers
8k views

Splitting train/test sets by an identifier?

I know sklearn has train_test_split() to split a train and test set. But I read that, even with setting a random seed, if your actual dataset is updated regularly, ...
Greg Rosen's user avatar
10 votes
1 answer
9k views

How do you calculate how dense or sparse a dataset is?

I'm looking deeper into collaborative filtering. One really interesting paper is "A Comparative Study of Collaborative Filtering Algorithms" http://arxiv.org/pdf/1205.3193.pdf In order to select ...
djones's user avatar
  • 203
10 votes
6 answers
1k views

What are some of the best practices for sharing data and models with colleagues?

As a data scientist who recently joined a new team, I wanted to ask the community how they share data and models among their colleagues. Currently I have to resort to storing data in some central ...
9 votes
4 answers
3k views

Suggest text classifier training datasets

Which freely available datasets can I use to train a text classifier? We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified ...
Abdelmawla's user avatar
9 votes
1 answer
24k views

How can I get the ImageNet ILSVRC 2012 data used for the classification challenge?

I would like to see if I can reproduce some of the image net results. However, I could not find the data (the list of URLs) used for training / testing in the ILSVRC 2012 (or later) classification ...
Martin Thoma's user avatar
  • 18.7k
9 votes
3 answers
387 views

Is (nearly) all data separable?

Suppose I have some data set with two classes. I could draw a decision boundary around each data point belonging to one of these classes, and hence, separate the data, like so: Where the red lines ...
Data's user avatar
  • 467
9 votes
3 answers
1k views

Interactive Graphing while logging data

I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to ...
Clayton Pipkin's user avatar
9 votes
2 answers
24k views

Always drop the first column after performing One Hot Encoding?

Since one of the columns can be generated completely from the others, and hence retaining this extra column does not add any new information for the modelling process, would it be good practice to ...
Gale's user avatar
  • 403
9 votes
2 answers
967 views

How to model user's buying behavior on Amazon?

For our final course project in Data Science, we proposed the following- Give the Amazon Reviews Dataset, we plan to come up with an algorithm (thats roughly based on Personalized PageRank) that ...
Pavan Manjunath's user avatar
8 votes
4 answers
10k views

One Hot encoding for large number of values

How do we use one hot encoding if the number of values which a categorical variable can take is large ? In my case it is 56 values. So as per usual method I would have to add 56 columns (56 binary ...
mach's user avatar
  • 367
8 votes
3 answers
3k views

What is normalization for?

I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do? ...
Pranit Kothari's user avatar
8 votes
1 answer
8k views

What is the distribution of categories in imagenet training set (ILSVRC2012)

http://arxiv.org/pdf/1409.0575v3.pdf Table 2 says there are 1,281,167 images and 732-1300 per class in the ILSVRC2012 training set. Ideally I'd like to avoid downloading the 138 GB just for this ...
user1030139's user avatar
8 votes
1 answer
14k views

Is it possible to have stratified train-test split of a set based on two columns?

Consider a dataframe that contains two columns, text and label. I can very easily create a stratified train-test split using ...
Aventinus's user avatar
  • 203
8 votes
4 answers
2k views

Job title similarity

I'm trying to define a metric between job titles in IT field. For this I need some metric between words of job titles that are not appearing together in the same job title, e.g. metric between the ...
Mher's user avatar
  • 181
8 votes
3 answers
8k views

Where can I find freely available multi-label datasets online? [closed]

I'm trying to find multi-label classfication datasets, which are available for free online. By "multi-label" I mean that each instance can be labeled with anywhere from a single to $k$ labels, where ...
Bobson Dugnutt's user avatar

1
2 3 4 5
30