Questions tagged [dataset]
A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData
52
questions
201
votes
35
answers
32k
views
Publicly Available Datasets
One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...
53
votes
5
answers
27k
views
Is it always better to use the whole dataset to train the final model?
A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, ...
- 3,880
59
votes
6
answers
16k
views
Should I go for a 'balanced' dataset or a 'representative' dataset?
My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
- 693
35
votes
4
answers
16k
views
Quick guide into training highly imbalanced data sets
I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
- 5,434
28
votes
3
answers
43k
views
Data Science Project Ideas [closed]
I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion.
I have just started with Data Science and ...
- 383
9
votes
3
answers
1k
views
Interactive Graphing while logging data
I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to ...
16
votes
3
answers
21k
views
When should we consider a dataset as imbalanced?
I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced.
My question is, are there any rules of thumb that tell us when we should subsample the large ...
- 584
11
votes
6
answers
28k
views
Python: Handling imbalance Classes in python Machine Learning
I have a dataset for which I am trying to predict target variables.
...
- 1,045
6
votes
2
answers
2k
views
What does it mean for the training data to be generated by a probability distribution over datasets
I was reading the Deep Learning book and came across the following para (page 109, second para):
The training and test data are generated by a probability distribution over datasets called the data-...
- 161
4
votes
2
answers
13k
views
How to classify and cluster this time series data [duplicate]
I have post already the question few months ago about my project that I'm starting to work on. This post can be see here:
Human activity recognition using smartphone data set problem
Now, I know ...
- 401
4
votes
1
answer
5k
views
How to interpret the loading values of a pca?
Imagine I've the following matrix, which gives the grades of students in the subjects German, Philosophy, Math and Physics:
...
- 281
3
votes
0
answers
294
views
Are SAS Data Storage Options designed for Big Data?
I understand that SAS has some data storage options in the form of
Default Storage tables or SAS datasets - in the SAS library.
Multi-dimensional Storage cubes - in the OLAP Server.
Third Party ...
- 795
28
votes
7
answers
27k
views
Publicly available social network datasets/APIs
As an extension to our great list of publicly available datasets, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside ...
- 4,097
21
votes
3
answers
10k
views
Dataset for Named Entity Recognition on Informal Text
I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the ...
- 2,029
17
votes
3
answers
5k
views
With unbalanced class, do I have to use under sampling on my validation/testing datasets?
I’m a beginner in machine learning and I’m facing a situation. I’m working on a Real Time Bidding problem, with the IPinYou dataset and I’m trying to do a click prediction.
The thing is that, as you ...
- 619
16
votes
2
answers
14k
views
How much data are sufficient to train my machine learning model?
I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining.
My colleague (who is a machine ...
- 281
13
votes
1
answer
4k
views
Do I have to standardize my new polynomial features?
I have a vector X with n features previously standardized.
If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
- 619
11
votes
2
answers
12k
views
How to split train/test in recommender systems
I am working with the MovieLens10M dataset, predicting user ratings. If I want to fairly evaluate my algorithm, how should I split my training v. test data?
By default, I believe the data is split ...
- 3,057
10
votes
1
answer
9k
views
How do you calculate how dense or sparse a dataset is?
I'm looking deeper into collaborative filtering. One really interesting paper is "A Comparative Study of Collaborative Filtering Algorithms" http://arxiv.org/pdf/1205.3193.pdf
In order to select ...
- 203
8
votes
3
answers
3k
views
What is normalization for?
I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do?
...
- 207
7
votes
2
answers
8k
views
For which real world data sets does DBSCAN surpass K-means.?
For clustering, DBSCAN surpass k-means in terms of handling arbitrary shape data sets. In the most published papers about density based clustering, the experiments are performed with synthetic data ...
- 237
7
votes
2
answers
11k
views
Why will the accuracy of a highly unbalanced dataset reduce after oversampling?
I have created a synthetic dataset, with 20 samples in one class and 100 in the other, thus creating an imbalanced dataset. Now the accuracy of classification of the data before balancing is 80% while ...
- 1,161
6
votes
1
answer
783
views
Should I prevent augmented data to leak to the test/cross validation sets
I have been working with the cats vs dogs dataset from kaggle which consist on 25000 images of cats and dogs labelled accordingly (btw, great dataset, totally recommended!)
One of the things I did ...
6
votes
1
answer
3k
views
Bechmark for Movielens
I'm looking for a place to find benchmarks against which to evaluate performance on public datasets.
In this instance, I'm interested in results on the MovieLens10M dataset. It seems to be ...
- 3,057
6
votes
3
answers
4k
views
Why is a correlation matrix symmetric?
I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix.
...
5
votes
2
answers
7k
views
Train-Test split for a recommender system
In all implementations of recommender systems I've seen so far, the train-test split is performed in this manner:
...
- 175
5
votes
1
answer
2k
views
Training and cross validation error curves
I have a graph which plots training datasize on X axis and accuracy on y axis. I plotted the curves using sklearn's learning_curve.
It is observed that the accuracy of training dataset decreases but ...
- 2,296
5
votes
4
answers
4k
views
Missing Values in Data [duplicate]
I have experienced that most of the datasets contain missing values, which make our task bit challenging.
Please let me know how to fill up those missing values in an efficient way? and is there any ...
- 339
4
votes
2
answers
2k
views
Is it correct to join training and validation set before inferring on test-set?
I would like to know if is a correct procedure to join training-set and validation-set together, in order to train the model on ...
- 693
4
votes
2
answers
12k
views
Ratio of positive to negative sample in data set for best classification
Suppose I have 100 positive samples. How many negative samples do I need to have in order to make the classifier work the best. In many papers, I have noticed that they take 4 times or 5 times the ...
- 1,161
4
votes
1
answer
771
views
How to predict based on multiple samples?
I am relatively new to ML so I apologies in advance if my question shows lack of understating of the field.
The problem
A particular study course has a high drop-out rate and we want to reduce it.
...
4
votes
3
answers
3k
views
What are the basic approaches for balancing a dataset for machine learning?
Sometimes we come across datasets where classes are imbalanced. For example, class A may have 2000 instances, but class B has only 200. How can we train a classifier for such datasets?
- 369
4
votes
1
answer
533
views
Training a classifier when some of the features are unknown
I am training a classifier in Matlab with a dataset that I created.
Unfortunately some of the features in the dataset were not recorded.
I currently have the unknown features set as -99999.
So, for ...
- 145
4
votes
1
answer
28k
views
Having trouble installing and loading tidyverse- No DIB package [closed]
after a fresh (Control panel, Windows 7) uninstall, then re-install of Rstudio, I have tried to install and load tidyverse. I get the following message, seemingly because the package DBI is missing:
...
- 41
3
votes
1
answer
5k
views
How to correctly perform data sampling for train/test split in multi-label dataset?
Problem statement
I have a text multi-label classification dataset, and I've found a problem with the dataset sampling.
I'm facing two different strategies. The first one consists in preprocessing ...
- 155
2
votes
4
answers
7k
views
Handling a feature containing multiple values
I have a dataset in following format:
Movie ID | Actors | Director | language | ReleaseYear | Genre
1 | Anil Kapoor;Manisha Koirala;Jackie Shroff;Anupam Kher;Danny Denzongpa;Pran | Vidhu Vinod ...
- 23
2
votes
2
answers
248
views
Dataset and why use evaluate()?
I am starting in Machine Learning, and I have doubts about some concepts. I've read we need to split our dataset into training, validation and test sets. I'll ask four questions related to them.
1 - ...
- 125
2
votes
2
answers
131
views
in which case can I say that the data are bad and I ll achieve nothing using machine learning on it
general Infos about my dataset: I have 40k data points and 5 features.
I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS ...
- 243
2
votes
2
answers
390
views
How to select features for a ML model
I have a dataset with 5K records for binary classification problem.
My features are min_blood_pressure, max_blood_pressure, <...
- 2,507
2
votes
1
answer
307
views
Missing at random vs missing not at random: What if it is both? (Does one imply the other?)
My understanding is that:
Missing at random: Whether or not a variable's value is missing is dependent on the values of the other variables.
Missing not at random: When the propensity for a variable'...
- 121
2
votes
2
answers
383
views
When a dataset is huge, what do you do to train with all the images on i t?
I'm using Python 3.7.7.
I'm trying to load a lot of NIFTI images using SimplyITK and Numpy from the [BraTS 2019 dataset][1].
This is the code I use to load the images into a numpy array.
...
- 229
2
votes
1
answer
121
views
How create a representative small subset from a huge dataset, for local development?
I have a time series problem and the dataset I'm using is rather huge. Around 100GB. For local development I'm trying to subset this into a very small batch around 50MB, just to make sure unit tests ...
- 123
2
votes
1
answer
215
views
Training a neural network with TWO possible correct outputs for one input
I have a system as a black box that has two correct outputs for a single input sample.
now I want to train a neural network to generate at least one of the correct outputs for that input sample.
what ...
2
votes
2
answers
6k
views
Prediction after one hot encoding
I have a regression model that I want to make prediction based on values that I will get from an end user.
In my dataset, I have one categorical variable region ...
- 33
2
votes
1
answer
2k
views
Why do train, test, validation datasets need to have the same distribution?
I've found a lot of martial on how to deal with differently distributed train/test/validation data sets, however I'm struggling to find out why they they need to have the same distribution.
Can ...
- 121
1
vote
2
answers
6k
views
Converting a nominal attributes to numerical ones in data set
I'm using the NSL-KDD data set which contains nominal and numerical values, and I want to convert all the nominal values to numerical ones. I tried the get_dummies ...
- 51
1
vote
3
answers
2k
views
Why do we scale down images before feeding them to the network?
I have been seeing a lot where images are generally scaled down to either $64\times64$, $32\times32$ or other lower resolutions. Can someone please help me with this and answer a few questions:
Don't ...
- 2,355
1
vote
1
answer
3k
views
Datasets for Topic Modeling [closed]
I'm looking to try and use deep learning methods for topic modeling as opposed to the more traditional methods of lda and word embedding methods. However, I'm having trouble finding good labeled ...
- 111
1
vote
0
answers
18
views
Non-parametric regression on set of time series: One model for each or one for all series?
Let's say I have a set of 1D time series which values have been samples in equip-distant time steps with timestamps $1,2,3,...$, they have all the same lengths and are somewhat similar in shape. I ...
- 752
0
votes
4
answers
1k
views
First steps on a new cleaned dataset
What is the very first thing you do when you get your hands on a new data set (assuming it is cleaned and well structured)? Please share sample code snippets as I am sure this would be extremely ...
- 397