Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
31 votes
1 answer
32k views

How is a splitting point chosen for continuous variables in decision trees?

I have two questions related to decision trees: If we have a continuous attribute, how do we choose the splitting value? Example: Age=(20,29,50,40....) Imagine that we have a continuous attribute $f$...
WALID BELRHALMIA's user avatar
20 votes
5 answers
17k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...
AffableAmbler's user avatar
4 votes
3 answers
471 views

Train classifier on balanced dataset and apply on imbalanced dataset?

I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works ...
user3240855's user avatar
1 vote
3 answers
300 views

Date transformation for KNN

I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful ...
Mapp's user avatar
  • 49
46 votes
2 answers
123k views

How does the validation_split parameter of Keras' fit function work?

Validation-split in Keras Sequential model fit function is documented as following on https://keras.io/models/sequential/ : validation_split: Float between 0 and 1. Fraction of the training data ...
rnso's user avatar
  • 1,558
16 votes
2 answers
15k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
DavideChicco.it's user avatar
13 votes
4 answers
19k views

Interpreting Decision Tree in context of feature importances

I'm trying to understand how to fully understand the decision process of a decision tree classification model built with sklearn. The 2 main aspect I'm looking at are a graphviz representation of the ...
Tim Lindsey's user avatar
13 votes
1 answer
4k views

Do I have to standardize my new polynomial features?

I have a vector X with n features previously standardized. If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
jmvllt's user avatar
  • 619
11 votes
2 answers
17k views

Oversampling/Undersampling only train set only or both train and validation set

I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set ...
yamini goel's user avatar
10 votes
5 answers
3k views

Tool to Generate 2D Data via Mouse Clicking

Often when I am learning new machine learning methods or experimenting with a data analysis algorithm I need to generate a series of 2D points. Teachers also do this often when making a lesson or ...
MD004's user avatar
  • 310
7 votes
2 answers
23k views

Merging large CSV files in pandas

I have two CSV files (each of the file size is in GBs) which I am trying to merge, but every time I do that, my computer hangs. Is there no way to merge them in chunks in pandas itself?
enterML's user avatar
  • 3,021
7 votes
6 answers
5k views

Generate timeseries data

Training would be bad if training data is not sufficient. Techniques like SMOTE or ADASYN can be used for oversampling. For image data, we can blur or change the angle to generate more samples from ...
vipin bansal's user avatar
  • 1,252
7 votes
6 answers
5k views

Is it advisable to combine two dataset?

I have two datasets on heart rate of subjects that were recorded in two different places (two different continent to be exact). The two research experiments aimed to find the subjects' emotions based ...
Lapatrie's user avatar
  • 145
6 votes
3 answers
4k views

Why is a correlation matrix symmetric?

I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix. ...
LazyAwkwardVaish Ok's user avatar
5 votes
4 answers
4k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
Abhishek Sharma's user avatar
5 votes
1 answer
3k views

Purpose of weights in neural networks

I'm beginner at Neural Networks. After reading multiple articles on wikipedia, i've seen the term "weight" being used a lot, although it is a little confusing. I know, that before the inputs are ...
ShellRox's user avatar
  • 409
5 votes
1 answer
3k views

Decision Tree used for Calculating Precision, Accuracy, and Recall, class breakdown question

I am creating decision trees modeling data that looks like this. ...
doedotdev's user avatar
  • 153
3 votes
2 answers
210 views

Domain-specific data science programs

I am looking for domain-specific data science programs (as a major, not a minor or specialization). I found programs in bio-statistics (and other disciplines in public health), MS in marketing ...
Hamideh's user avatar
  • 940
3 votes
2 answers
2k views

Do i need to handle missing values before EDA?

I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to ...
José Augusto's user avatar
3 votes
2 answers
835 views

How to handle non ordinal Features like Gender,Language,Region etc? Ordinal Encoding or one-hot encoding?

I see that usually, while preparing the dataset. Usually, data scientists convert non-ordinal features like Gender or Language in a dataset using LabelEncoder/ordinalEncoder. Ideally, they should have ...
Nitin Shravan's user avatar
2 votes
1 answer
824 views

SMOTE oversampling for class imbalanced dataset introduces bias in final distribution

I have a problem statement where percentage of goods (denoted by 0) is 95%, and for bads (denoted by 1) it is 5% only. One way is to do under sampling of goods so that model understands the patterns ...
Deepak's user avatar
  • 217
2 votes
2 answers
134 views

in which case can I say that the data are bad and I ll achieve nothing using machine learning on it

general Infos about my dataset: I have 40k data points and 5 features. I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS ...
basilisk's user avatar
  • 243
2 votes
2 answers
333 views

Data Imbalance in Regression Tasks [closed]

Having read a lot about class imbalance in classification tasks, I'd like to know what is the methodology for data imbalance in regression tasks. Particularly, - What is the procedure to check for ...
Pepe's user avatar
  • 123
2 votes
2 answers
700 views

Best Programming Language for Data Science [closed]

I'm learning JS, HTML and CSS, but I doubt JS is very good at Data Analysis. So, what would you guys recommend me learning to start my "career" in Data Science? What's the best programming language ...
Ben's user avatar
  • 123
2 votes
1 answer
184 views

How create a representative small subset from a huge dataset, for local development?

​ I have a time series problem and the dataset I'm using is rather huge. Around 100GB. For local development I'm trying to subset this into a very small batch around 50MB, just to make sure unit tests ...
Farhood ET's user avatar
1 vote
2 answers
127 views

Problem with converting string to dummy variables

I'm new in data science, I have data which want to work on it, I omitted extra columns and convert it to 4 columns ( Product, Date, Market, Demand ) . in this data Product and Market are string, I ...
ramin's user avatar
  • 13
1 vote
1 answer
425 views

Computing Jaccard Similarity between two documents

Data Mining: Compute the Jaccard similarity of D1 and D2 on 2-shingles is. Sim(D1,D2) = D1 = the quick brown fox jumps over the lazy dog D2 = jeff typed the quick brown dog jumps over the lazy fox ...
Danniy's user avatar
  • 13
1 vote
1 answer
175 views

Best practice - memory management when data wrangling

I was wondering what are the best resources to learn the best practice when it comes to memory management. For example, lets say I have the following code below: ...
wombatkingdom's user avatar
1 vote
1 answer
2k views

Data augmentation for multiple output heads in Keras

I have a transfer learning based two output classification problem. So, accordingly, I have formatted my data to have X_train as a ...
Prasad Raghavendra's user avatar
0 votes
1 answer
115 views

Image data augmentation with this function

Following function is from http://scikit-learn.org/stable/auto_examples/neural_networks/plot_rbm_logistic_classification.html#sphx-glr-auto-examples-neural-networks-plot-rbm-logistic-classification-py ...
rnso's user avatar
  • 1,558
0 votes
2 answers
140 views

What is valued more in the data science job market, statistical analysis or data processing?

Not sure if this question is OT here. If it is, perhaps move it to meta? If not: I'm trained in classical statistics, work in research, and have learned much (and taught) machine learning. I've ...
generic_user's user avatar
0 votes
1 answer
157 views

How to distort data in a clever way? [closed]

For example, I have some time series. How can I change my data, so it will not be obvious to understand what was original values? Ideally transformation would allow to revert and reconstruct ...
StochasticIntegrationStudent's user avatar
0 votes
1 answer
315 views

Can we have a dataset with slight difference in target values for same value of feature variable?

I am trying to generate a dataset which involves 1 feature variable(X) and 1 target variable(y). The feature variable ...
Deepak Tatyaji Ahire's user avatar