Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
202 votes
35 answers
33k views

Publicly Available Datasets

One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...
55 votes
5 answers
32k views

Is it always better to use the whole dataset to train the final model?

A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, ...
pcko1's user avatar
  • 3,930
59 votes
6 answers
16k views

Should I go for a 'balanced' dataset or a 'representative' dataset?

My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
pnp's user avatar
  • 693
35 votes
4 answers
16k views

Quick guide into training highly imbalanced data sets

I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
IgorS's user avatar
  • 5,464
28 votes
3 answers
43k views

Data Science Project Ideas [closed]

I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion. I have just started with Data Science and ...
Kevin Desai's user avatar
9 votes
3 answers
1k views

Interactive Graphing while logging data

I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to ...
Clayton Pipkin's user avatar
17 votes
3 answers
23k views

When should we consider a dataset as imbalanced?

I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced. My question is, are there any rules of thumb that tell us when we should subsample the large ...
Rami's user avatar
  • 594
11 votes
6 answers
28k views

Python: Handling imbalance Classes in python Machine Learning

I have a dataset for which I am trying to predict target variables. ...
SRS's user avatar
  • 1,045
6 votes
2 answers
2k views

What does it mean for the training data to be generated by a probability distribution over datasets

I was reading the Deep Learning book and came across the following para (page 109, second para): The training and test data are generated by a probability distribution over datasets called the data-...
humble's user avatar
  • 161
4 votes
1 answer
5k views

How to interpret the loading values of a pca?

Imagine I've the following matrix, which gives the grades of students in the subjects German, Philosophy, Math and Physics: ...
So S's user avatar
  • 281
4 votes
2 answers
13k views

How to classify and cluster this time series data [duplicate]

I have post already the question few months ago about my project that I'm starting to work on. This post can be see here: Human activity recognition using smartphone data set problem Now, I know ...
Jakubee's user avatar
  • 401
3 votes
0 answers
301 views

Are SAS Data Storage Options designed for Big Data?

I understand that SAS has some data storage options in the form of Default Storage tables or SAS datasets - in the SAS library. Multi-dimensional Storage cubes - in the OLAP Server. Third Party ...
Minu's user avatar
  • 795
28 votes
7 answers
27k views

Publicly available social network datasets/APIs

As an extension to our great list of publicly available datasets, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside ...
Rubens's user avatar
  • 4,107
21 votes
3 answers
10k views

Dataset for Named Entity Recognition on Informal Text

I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the ...
Madison May's user avatar
  • 2,029
17 votes
3 answers
6k views

With unbalanced class, do I have to use under sampling on my validation/testing datasets?

I’m a beginner in machine learning and I’m facing a situation. I’m working on a Real Time Bidding problem, with the IPinYou dataset and I’m trying to do a click prediction. The thing is that, as you ...
jmvllt's user avatar
  • 619
16 votes
2 answers
15k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
DavideChicco.it's user avatar
13 votes
1 answer
4k views

Do I have to standardize my new polynomial features?

I have a vector X with n features previously standardized. If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
jmvllt's user avatar
  • 619
12 votes
2 answers
12k views

How to split train/test in recommender systems

I am working with the MovieLens10M dataset, predicting user ratings. If I want to fairly evaluate my algorithm, how should I split my training v. test data? By default, I believe the data is split ...
jamesmf's user avatar
  • 3,087
10 votes
1 answer
10k views

How do you calculate how dense or sparse a dataset is?

I'm looking deeper into collaborative filtering. One really interesting paper is "A Comparative Study of Collaborative Filtering Algorithms" http://arxiv.org/pdf/1205.3193.pdf In order to select ...
djones's user avatar
  • 203
8 votes
3 answers
3k views

What is normalization for?

I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do? ...
Pranit Kothari's user avatar
7 votes
2 answers
11k views

Why will the accuracy of a highly unbalanced dataset reduce after oversampling?

I have created a synthetic dataset, with 20 samples in one class and 100 in the other, thus creating an imbalanced dataset. Now the accuracy of classification of the data before balancing is 80% while ...
girl101's user avatar
  • 1,161
7 votes
2 answers
9k views

For which real world data sets does DBSCAN surpass K-means.?

For clustering, DBSCAN surpass k-means in terms of handling arbitrary shape data sets. In the most published papers about density based clustering, the experiments are performed with synthetic data ...
tanawatl's user avatar
  • 237
6 votes
1 answer
3k views

Bechmark for Movielens

I'm looking for a place to find benchmarks against which to evaluate performance on public datasets. In this instance, I'm interested in results on the MovieLens10M dataset. It seems to be ...
jamesmf's user avatar
  • 3,087
6 votes
3 answers
4k views

Why is a correlation matrix symmetric?

I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix. ...
LazyAwkwardVaish Ok's user avatar
6 votes
1 answer
858 views

Should I prevent augmented data to leak to the test/cross validation sets

I have been working with the cats vs dogs dataset from kaggle which consist on 25000 images of cats and dogs labelled accordingly (btw, great dataset, totally recommended!) One of the things I did ...
Juan Antonio Gomez Moriano's user avatar
5 votes
2 answers
2k views

Is it correct to join training and validation set before inferring on test-set?

I would like to know if is a correct procedure to join training-set and validation-set together, in order to train the model on ...
Simone's user avatar
  • 705
5 votes
4 answers
4k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
Abhishek Sharma's user avatar
5 votes
1 answer
2k views

Training and cross validation error curves

I have a graph which plots training datasize on X axis and accuracy on y axis. I plotted the curves using sklearn's learning_curve. It is observed that the accuracy of training dataset decreases but ...
Hima Varsha's user avatar
  • 2,316
5 votes
2 answers
8k views

Train-Test split for a recommender system

In all implementations of recommender systems I've seen so far, the train-test split is performed in this manner: ...
Qubix's user avatar
  • 175
4 votes
1 answer
28k views

Having trouble installing and loading tidyverse- No DIB package [closed]

after a fresh (Control panel, Windows 7) uninstall, then re-install of Rstudio, I have tried to install and load tidyverse. I get the following message, seemingly because the package DBI is missing: ...
jshea's user avatar
  • 41
4 votes
3 answers
3k views

What are the basic approaches for balancing a dataset for machine learning?

Sometimes we come across datasets where classes are imbalanced. For example, class A may have 2000 instances, but class B has only 200. How can we train a classifier for such datasets?
10land's user avatar
  • 369
4 votes
2 answers
12k views

Ratio of positive to negative sample in data set for best classification

Suppose I have 100 positive samples. How many negative samples do I need to have in order to make the classifier work the best. In many papers, I have noticed that they take 4 times or 5 times the ...
girl101's user avatar
  • 1,161
4 votes
1 answer
926 views

How to predict based on multiple samples?

I am relatively new to ML so I apologies in advance if my question shows lack of understating of the field. The problem A particular study course has a high drop-out rate and we want to reduce it. ...
Alessandro Di Bella's user avatar
4 votes
1 answer
621 views

Training a classifier when some of the features are unknown

I am training a classifier in Matlab with a dataset that I created. Unfortunately some of the features in the dataset were not recorded. I currently have the unknown features set as -99999. So, for ...
DarkLink's user avatar
  • 145
3 votes
1 answer
3k views

Why do train, test, validation datasets need to have the same distribution?

I've found a lot of martial on how to deal with differently distributed train/test/validation data sets, however I'm struggling to find out why they they need to have the same distribution. Can ...
zaza's user avatar
  • 131
3 votes
1 answer
5k views

How to correctly perform data sampling for train/test split in multi-label dataset?

Problem statement I have a text multi-label classification dataset, and I've found a problem with the dataset sampling. I'm facing two different strategies. The first one consists in preprocessing ...
Alber8295's user avatar
  • 155
2 votes
2 answers
433 views

When a dataset is huge, what do you do to train with all the images on i t?

I'm using Python 3.7.7. I'm trying to load a lot of NIFTI images using SimplyITK and Numpy from the [BraTS 2019 dataset][1]. This is the code I use to load the images into a numpy array. ...
VansFannel's user avatar
2 votes
1 answer
393 views

Missing at random vs missing not at random: What if it is both? (Does one imply the other?)

My understanding is that: Missing at random: Whether or not a variable's value is missing is dependent on the values of the other variables. Missing not at random: When the propensity for a variable'...
Edward Garemo's user avatar
2 votes
1 answer
234 views

How create a representative small subset from a huge dataset, for local development?

​ I have a time series problem and the dataset I'm using is rather huge. Around 100GB. For local development I'm trying to subset this into a very small batch around 50MB, just to make sure unit tests ...
Farhood ET's user avatar
2 votes
2 answers
340 views

Dataset and why use evaluate()?

I am starting in Machine Learning, and I have doubts about some concepts. I've read we need to split our dataset into training, validation and test sets. I'll ask four questions related to them. 1 - ...
Murilo's user avatar
  • 125
2 votes
4 answers
7k views

Handling a feature containing multiple values

I have a dataset in following format: Movie ID | Actors | Director | language | ReleaseYear | Genre 1 | Anil Kapoor;Manisha Koirala;Jackie Shroff;Anupam Kher;Danny Denzongpa;Pran | Vidhu Vinod ...
romitheguru's user avatar
2 votes
1 answer
278 views

Training a neural network with TWO possible correct outputs for one input

I have a system as a black box that has two correct outputs for a single input sample. now I want to train a neural network to generate at least one of the correct outputs for that input sample. what ...
Abolfazl Sajady's user avatar
2 votes
2 answers
464 views

How to select features for a ML model

I have a dataset with 5K records for binary classification problem. My features are min_blood_pressure, max_blood_pressure, <...
The Great's user avatar
  • 2,565
2 votes
2 answers
135 views

in which case can I say that the data are bad and I ll achieve nothing using machine learning on it

general Infos about my dataset: I have 40k data points and 5 features. I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS ...
basilisk's user avatar
  • 243
2 votes
2 answers
7k views

Prediction after one hot encoding

I have a regression model that I want to make prediction based on values that I will get from an end user. In my dataset, I have one categorical variable region ...
IngridX's user avatar
  • 33
1 vote
0 answers
20 views

Non-parametric regression on set of time series: One model for each or one for all series?

Let's say I have a set of 1D time series which values have been samples in equip-distant time steps with timestamps $1,2,3,...$, they have all the same lengths and are somewhat similar in shape. I ...
Make42's user avatar
  • 752
1 vote
1 answer
3k views

Datasets for Topic Modeling [closed]

I'm looking to try and use deep learning methods for topic modeling as opposed to the more traditional methods of lda and word embedding methods. However, I'm having trouble finding good labeled ...
David's user avatar
  • 111
1 vote
2 answers
7k views

Converting a nominal attributes to numerical ones in data set

I'm using the NSL-KDD data set which contains nominal and numerical values, and I want to convert all the nominal values to numerical ones. I tried the get_dummies ...
user4309930's user avatar
1 vote
3 answers
2k views

Why do we scale down images before feeding them to the network?

I have been seeing a lot where images are generally scaled down to either $64\times64$, $32\times32$ or other lower resolutions. Can someone please help me with this and answer a few questions: Don't ...
thanatoz's user avatar
  • 2,395
0 votes
4 answers
1k views

First steps on a new cleaned dataset

What is the very first thing you do when you get your hands on a new data set (assuming it is cleaned and well structured)? Please share sample code snippets as I am sure this would be extremely ...
Sidhha's user avatar
  • 397