Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
174
votes
35answers
28k views

Publicly Available Datasets

One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...
49
votes
5answers
11k views

Should I go for a 'balanced' dataset or a 'representative' dataset?

My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
29
votes
4answers
13k views

Quick guide into training highly imbalanced data sets

I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
25
votes
3answers
42k views

Data Science Project Ideas [closed]

I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion. I have just started with Data Science and ...
27
votes
4answers
10k views

Is it always better to use the whole dataset to train the final model?

A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, ...
8
votes
3answers
955 views

Interactive Graphing while logging data

I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to ...
10
votes
3answers
12k views

When should we consider a dataset as imbalanced?

I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced. My question is, are there any rules of thumb that tell us when we should subsample the large ...
9
votes
6answers
23k views

Python: Handling imbalance Classes in python Machine Learning

I have a dataset for which I am trying to predict target variables. ...
3
votes
3answers
11k views

How to classify and cluster this time series data

I have post already the question few months ago about my project that I'm starting to work on. This post can be see here: Human activity recognition using smartphone data set problem Now, I know ...
4
votes
2answers
637 views

What does it mean for the training data to be generated by a probability distribution over datasets

I was reading the Deep Learning book and came across the following para (page 109, second para): The training and test data are generated by a probability distribution over datasets called the data-...
4
votes
0answers
242 views

Are SAS Data Storage Options designed for Big Data?

I understand that SAS has some data storage options in the form of Default Storage tables or SAS datasets - in the SAS library. Multi-dimensional Storage cubes - in the OLAP Server. Third Party ...
4
votes
1answer
3k views

How to interpret the loading values of a pca?

Imagine I've the following matrix, which gives the grades of students in the subjects German, Philosophy, Math and Physics: ...
26
votes
7answers
26k views

Publicly available social network datasets/APIs

As an extension to our great list of publicly available datasets, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside ...
14
votes
3answers
4k views

With unbalanced class, do I have to use under sampling on my validation/testing datasets?

I’m a beginner in machine learning and I’m facing a situation. I’m working on a Real Time Bidding problem, with the IPinYou dataset and I’m trying to do a click prediction. The thing is that, as you ...
11
votes
2answers
8k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
18
votes
3answers
8k views

Dataset for Named Entity Recognition on Informal Text

I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the ...
6
votes
2answers
5k views

For which real world data sets does DBSCAN surpass K-means.?

For clustering, DBSCAN surpass k-means in terms of handling arbitrary shape data sets. In the most published papers about density based clustering, the experiments are performed with synthetic data ...
5
votes
4answers
3k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
5
votes
1answer
11k views

Always drop the first column after performing One Hot Encoding?

Since one of the columns can be generated completely from the others, and hence retaining this extra column does not add any new information for the modelling process, would it be good practice to ...
4
votes
1answer
2k views

Training and cross validation error curves

I have a graph which plots training datasize on X axis and accuracy on y axis. I plotted the curves using sklearn's learning_curve. It is observed that the accuracy of training dataset decreases but ...
6
votes
1answer
3k views

Bechmark for Movielens

I'm looking for a place to find benchmarks against which to evaluate performance on public datasets. In this instance, I'm interested in results on the MovieLens10M dataset. It seems to be ...
4
votes
1answer
25k views

Having trouble installing and loading tidyverse- No DIB package [closed]

after a fresh (Control panel, Windows 7) uninstall, then re-install of Rstudio, I have tried to install and load tidyverse. I get the following message, seemingly because the package DBI is missing: ...
3
votes
2answers
8k views

Ratio of positive to negative sample in data set for best classification

Suppose I have 100 positive samples. How many negative samples do I need to have in order to make the classifier work the best. In many papers, I have noticed that they take 4 times or 5 times the ...
2
votes
2answers
180 views

How to select features for a ML model

I have a dataset with 5K records for binary classification problem. My features are min_blood_pressure, max_blood_pressure, <...
1
vote
4answers
903 views

First steps on a new cleaned dataset

What is the very first thing you do when you get your hands on a new data set (assuming it is cleaned and well structured)? Please share sample code snippets as I am sure this would be extremely ...
6
votes
1answer
1k views

Do I have to standardize my new polynomial features?

I have a vector X with n features previously standardized. If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
4
votes
2answers
2k views

What are the basic approaches for balancing a dataset for machine learning?

Sometimes we come across datasets where classes are imbalanced. For example, class A may have 2000 instances, but class B has only 200. How can we train a classifier for such datasets?
4
votes
2answers
5k views

Why will the accuracy of a highly unbalanced dataset reduce after oversampling?

I have created a synthetic dataset, with 20 samples in one class and 100 in the other, thus creating an imbalanced dataset. Now the accuracy of classification of the data before balancing is 80% while ...
1
vote
2answers
4k views

Converting a nominal attributes to numerical ones in data set

I'm using the NSL-KDD data set which contains nominal and numerical values, and I want to convert all the nominal values to numerical ones. I tried the get_dummies ...
6
votes
3answers
3k views

What is normalization for?

I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do? ...
4
votes
2answers
66 views

Discrimination vs Calibration - Machine Learning Models

I came across a new term called Calibration while reading about prediction models. Can you please help me understand how different it is from ...
4
votes
2answers
1k views

Why is a correlation matrix symmetric?

I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix. ...
3
votes
1answer
106 views

How to predict based on multiple samples?

I am relatively new to ML so I apologies in advance if my question shows lack of understating of the field. The problem A particular study course has a high drop-out rate and we want to reduce it. ...
2
votes
2answers
86 views

in which case can I say that the data are bad and I ll achieve nothing using machine learning on it

general Infos about my dataset: I have 40k data points and 5 features. I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS ...
1
vote
0answers
10 views

Non-parametric regression on set of time series: One model for each or one for all series?

Let's say I have a set of 1D time series which values have been samples in equip-distant time steps with timestamps $1,2,3,...$, they have all the same lengths and are somewhat similar in shape. I ...
1
vote
1answer
283 views

Datasets for Topic Modeling [closed]

I'm looking to try and use deep learning methods for topic modeling as opposed to the more traditional methods of lda and word embedding methods. However, I'm having trouble finding good labeled ...
1
vote
3answers
102 views

Why do we scale down images before feeding them to the network?

I have been seeing a lot where images are generally scaled down to either $64\times64$, $32\times32$ or other lower resolutions. Can someone please help me with this and answer a few questions: Don't ...
0
votes
2answers
1k views

Prediction after one hot encoding

I have a regression model that I want to make prediction based on values that I will get from an end user. In my dataset, I have one categorical variable region ...
0
votes
0answers
796 views

How to map RGB image segmentation ground truth to classes/one-hot vectors in TensorFlow?

Is there a TensorFlow function which takes multicategorical ground truth for semantic image segmentations in the form of RGB images and outputs a tensor with a one-hot vector encoding of the ...