Questions tagged [dataset]

A dataset is a collection of data, often in tabular or matrix form. This tag is NOT intended for data requests ("where can I find a dataset about ...") --> see OpenData

Filter by
Sorted by
Tagged with
173
votes
35answers
28k views

Publicly Available Datasets

One of the common problems in data science is gathering data from various sources in a somehow cleaned (semi-structured) format and combining metrics from various sources for making a higher level ...
48
votes
5answers
11k views

Should I go for a 'balanced' dataset or a 'representative' dataset?

My 'machine learning' task is of separating benign Internet traffic from malicious traffic. In the real world scenario, most (say 90% or more) of Internet traffic is benign. Thus I felt that I should ...
29
votes
4answers
12k views

Quick guide into training highly imbalanced data sets

I have a classification problem with approximately 1000 positive and 10000 negative samples in training set. So this data set is quite unbalanced. Plain random forest is just trying to mark all test ...
24
votes
3answers
42k views

Data Science Project Ideas [closed]

I don't know if this is a right place to ask this question, but a community dedicated to Data Science should be the most appropriate place in my opinion. I have just started with Data Science and ...
8
votes
3answers
950 views

Interactive Graphing while logging data

I'm looking to graph and interactively explore live/continuously measured data. There are quite a few options out there, with plot.ly being the most user-friendly. Plot.ly has a fantastic and easy to ...
10
votes
3answers
12k views

When should we consider a dataset as imbalanced?

I'm facing a situation where the numbers of positive and negative examples in a dataset are imbalanced. My question is, are there any rules of thumb that tell us when we should subsample the large ...
9
votes
6answers
22k views

Python: Handling imbalance Classes in python Machine Learning

I have a dataset for which I am trying to predict target variables. ...
3
votes
3answers
10k views

How to classify and cluster this time series data

I have post already the question few months ago about my project that I'm starting to work on. This post can be see here: Human activity recognition using smartphone data set problem Now, I know ...
4
votes
2answers
582 views

What does it mean for the training data to be generated by a probability distribution over datasets

I was reading the Deep Learning book and came across the following para (page 109, second para): The training and test data are generated by a probability distribution over datasets called the data-...
4
votes
0answers
238 views

Are SAS Data Storage Options designed for Big Data?

I understand that SAS has some data storage options in the form of Default Storage tables or SAS datasets - in the SAS library. Multi-dimensional Storage cubes - in the OLAP Server. Third Party ...
4
votes
1answer
3k views

How to interpret the loading values of a pca?

Imagine I've the following matrix, which gives the grades of students in the subjects German, Philosophy, Math and Physics: ...
26
votes
7answers
25k views

Publicly available social network datasets/APIs

As an extension to our great list of publicly available datasets, I'd like to know if there is any list of publicly available social network datasets/crawling APIs. It would be very nice if alongside ...
26
votes
4answers
9k views

Is it always better to use the whole dataset to train the final model?

A common technique after training, validating and testing the Machine Learning model of preference is to use the complete dataset, including the testing subset, to train a final model to deploy it on, ...
11
votes
2answers
8k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
18
votes
3answers
8k views

Dataset for Named Entity Recognition on Informal Text

I'm currently searching for labeled datasets to train a model to extract named entities from informal text (something similar to tweets). Because capitalization and grammar are often lacking in the ...
6
votes
2answers
5k views

For which real world data sets does DBSCAN surpass K-means.?

For clustering, DBSCAN surpass k-means in terms of handling arbitrary shape data sets. In the most published papers about density based clustering, the experiments are performed with synthetic data ...
5
votes
4answers
3k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
4
votes
1answer
2k views

Training and cross validation error curves

I have a graph which plots training datasize on X axis and accuracy on y axis. I plotted the curves using sklearn's learning_curve. It is observed that the accuracy of training dataset decreases but ...
6
votes
1answer
3k views

Bechmark for Movielens

I'm looking for a place to find benchmarks against which to evaluate performance on public datasets. In this instance, I'm interested in results on the MovieLens10M dataset. It seems to be ...
4
votes
1answer
24k views

Having trouble installing and loading tidyverse- No DIB package [closed]

after a fresh (Control panel, Windows 7) uninstall, then re-install of Rstudio, I have tried to install and load tidyverse. I get the following message, seemingly because the package DBI is missing: ...
3
votes
2answers
8k views

Ratio of positive to negative sample in data set for best classification

Suppose I have 100 positive samples. How many negative samples do I need to have in order to make the classifier work the best. In many papers, I have noticed that they take 4 times or 5 times the ...
2
votes
2answers
171 views

How to select features for a ML model

I have a dataset with 5K records for binary classification problem. My features are min_blood_pressure, max_blood_pressure, <...
1
vote
4answers
896 views

First steps on a new cleaned dataset

What is the very first thing you do when you get your hands on a new data set (assuming it is cleaned and well structured)? Please share sample code snippets as I am sure this would be extremely ...
6
votes
1answer
1k views

Do I have to standardize my new polynomial features?

I have a vector X with n features previously standardized. If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
4
votes
2answers
2k views

What are the basic approaches for balancing a dataset for machine learning?

Sometimes we come across datasets where classes are imbalanced. For example, class A may have 2000 instances, but class B has only 200. How can we train a classifier for such datasets?
1
vote
2answers
4k views

Converting a nominal attributes to numerical ones in data set

I'm using the NSL-KDD data set which contains nominal and numerical values, and I want to convert all the nominal values to numerical ones. I tried the get_dummies ...
6
votes
3answers
3k views

What is normalization for?

I am new in python and data science (and not great in math). I am learning machine learning. I got following normalize function. Can you please explain what does this normalize function do? ...
4
votes
2answers
1k views

Why is a correlation matrix symmetric?

I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix. ...
3
votes
1answer
73 views

How to predict based on multiple samples?

I am relatively new to ML so I apologies in advance if my question shows lack of understating of the field. The problem A particular study course has a high drop-out rate and we want to reduce it. ...
2
votes
2answers
81 views

in which case can I say that the data are bad and I ll achieve nothing using machine learning on it

general Infos about my dataset: I have 40k data points and 5 features. I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS ...
1
vote
1answer
155 views

Datasets for Topic Modeling [closed]

I'm looking to try and use deep learning methods for topic modeling as opposed to the more traditional methods of lda and word embedding methods. However, I'm having trouble finding good labeled ...
1
vote
3answers
87 views

Why do we scale down images before feeding them to the network?

I have been seeing a lot where images are generally scaled down to either $64\times64$, $32\times32$ or other lower resolutions. Can someone please help me with this and answer a few questions: Don't ...
1
vote
0answers
9 views

Non-parametric regression on set of time series: One model for each or one for all series?

Let's say I have a set of 1D time series which values have been samples in equip-distant time steps with timestamps $1,2,3,...$, they have all the same lengths and are somewhat similar in shape. I ...
0
votes
0answers
709 views

How to map RGB image segmentation ground truth to classes/one-hot vectors in TensorFlow?

Is there a TensorFlow function which takes multicategorical ground truth for semantic image segmentations in the form of RGB images and outputs a tensor with a one-hot vector encoding of the ...
0
votes
2answers
816 views

Prediction after one hot encoding

I have a regression model that I want to make prediction based on values that I will get from an end user. In my dataset, I have one categorical variable region ...