Questions tagged [data]

Questions mostly concerned with managing data, without focus on pre-processing or modelling.

Filter by
Sorted by
Tagged with
19
votes
1answer
20k views

How is a splitting point chosen for continuous variables in decision trees?

I have two questions related to decision trees: If we have a continuous attribute, how do we choose the splitting value? Example: Age=(20,29,50,40....) Imagine that we have a continuous attribute $f$...
15
votes
5answers
13k views

Do modern R and/or Python libraries make SQL obsolete?

I work in an office where SQL Server is the backbone of everything we do, from data processing to cleaning to munging. My colleague specializes in writing complex functions and stored procedures to ...
0
votes
2answers
126 views

Date transformation for KNN

I have data set with date features like 01/01/2019 and I would like to use KNN. However, I cannot find a good transformation for dates that has a meaningful ...
12
votes
2answers
10k views

How much data are sufficient to train my machine learning model?

I've been working on machine learning and bioinformatics for a while, and today I had a conversation with a colleague about the main general issues of data mining. My colleague (who is a machine ...
9
votes
4answers
15k views

Interpreting Decision Tree in context of feature importances

I'm trying to understand how to fully understand the decision process of a decision tree classification model built with sklearn. The 2 main aspect I'm looking at are a graphviz representation of the ...
8
votes
1answer
2k views

Do I have to standardize my new polynomial features?

I have a vector X with n features previously standardized. If I want to generate new polynomial features (let say adding square features), do I need to do another standardization on these new ...
5
votes
4answers
3k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
5
votes
2answers
16k views

Merging large CSV files in Pandas

I have two CSV files(each of the file size is in GBs). I am trying to merge the two CSV files but every time I try to it my computer hangs. Is there no way to merge the files in chunks in pandas ...
8
votes
6answers
2k views

Generate timeseries data

Training would be bad if training data is not sufficient. Techniques like SMOTE or ADASYN can be used for oversampling. For image data, we can blur or change the angle to generate more samples from ...
5
votes
1answer
2k views

Decision Tree used for Calculating Precision, Accuracy, and Recall, class breakdown question

I am creating decision trees modeling data that looks like this. ...
5
votes
1answer
2k views

Purpose of weights in neural networks

I'm beginner at Neural Networks. After reading multiple articles on wikipedia, i've seen the term "weight" being used a lot, although it is a little confusing. I know, that before the inputs are ...
4
votes
3answers
349 views

Train classifier on balanced dataset and apply on imbalanced dataset?

I have a labelled training dataset DS1 with 1000 entries. The targets (True/False) are nearly balanced. With sklearn, I have tried several algorithms, of which the GradientBoostingClassifier works ...
2
votes
1answer
839 views

How to extract the sample split (values) of decision tree leaves ( terminal nodes) applying h2o library

Sorry for a long story, but it is a long story. :) I am using h2o library for python to build a decision tree and to extract a decision rules out of it. I use some data for training where labels get ...
6
votes
3answers
2k views

Why is a correlation matrix symmetric?

I'm sorry for being so weak in math. (I'm a student) For eg. this is a correlation matrix. ...
4
votes
2answers
2k views

Oversampling/Undersampling only train set only or both train and validation set

I am working on a dataset with class imbalance problem. Now, I know one needs to oversample or undersample only the train set and not the test set. But my issue is: whether to oversample the train set ...
3
votes
2answers
189 views

Domain-specific data science programs

I am looking for domain-specific data science programs (as a major, not a minor or specialization). I found programs in bio-statistics (and other disciplines in public health), MS in marketing ...
2
votes
2answers
94 views

in which case can I say that the data are bad and I ll achieve nothing using machine learning on it

general Infos about my dataset: I have 40k data points and 5 features. I'm doing regression and trying to build a model that can predict the error of a GPS. for example imagine that your vehicle GPS ...
0
votes
1answer
58 views

Can we have a dataset with slight difference in target values for same value of feature variable?

I am trying to generate a dataset which involves 1 feature variable(X) and 1 target variable(y). The feature variable ...
0
votes
1answer
76 views

Image data augmentation with this function

Following function is from http://scikit-learn.org/stable/auto_examples/neural_networks/plot_rbm_logistic_classification.html#sphx-glr-auto-examples-neural-networks-plot-rbm-logistic-classification-py ...
0
votes
1answer
269 views

Data augmentation for multiple output heads in Keras

I have a transfer learning based two output classification problem. So, accordingly, I have formatted my data to have X_train as a ...
0
votes
1answer
125 views

How to distort data in a clever way? [closed]

For example, I have some time series. How can I change my data, so it will not be obvious to understand what was original values? Ideally transformation would allow to revert and reconstruct ...