48
votes
Difference between OrdinalEncoder and LabelEncoder
Afaik, both have the same functionality. A bit difference is the idea behind. OrdinalEncoder is for converting features, while ...
- 816
46
votes
Accepted
StandardScaler before or after splitting data - which is better?
In the interest of preventing information about the distribution of the test set leaking into your model, you should go for option #2 and fit the scaler on your training data only, then standardise ...
- 1,668
34
votes
Accepted
How to prepare/augment images for neural network?
The idea with Neural Networks is that they need little pre-processing since the heavy lifting is done by the algorithm which is the one in charge of learning the features.
The winners of the Data ...
- 3,390
26
votes
Accepted
One Hot Encoding vs Word Embedding - When to choose one or another?
One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each ...
- 20.4k
26
votes
Difference between OrdinalEncoder and LabelEncoder
As for differences in OrdinalEncoder and LabelEncoder implementation, the accepted answer mentions the shape of the data:
...
- 393
24
votes
Different Test Set and Training Set Distribution
Great question, this is what is known in Machine Learning paradigm as either
"Covariate Shift", or "Model Drift" or "Nonstationarity" and so on.
One of the critical assumption one would make to ...
- 4,207
18
votes
Accepted
Image resizing and padding for CNN
This question on stackoverflow might help you. To sum up, some deep learning researchers think that padding a big part of the image is not a good practice, since the neural network has to learn that ...
- 6,031
18
votes
Accepted
In ML why selecting the best variables?
You are right. If someone is using regularization correctly and doing hyperparameter tuning to avoid overfitting, then it should not be a problem theoretically (ie multi-collinearity will not reduce ...
- 805
16
votes
Loading own train data and labels in dataloader using pytorch?
Assuming both of x_data and labels are lists or numpy arrays,
...
- 260
15
votes
Accepted
Encoding with OrdinalEncoder : how to give levels as user input?
I'm not sure if you ever figured this out but I was trying to find answers on this exact same question and there aren't really any good answers in my opinion. I finally figured it out though.
...
- 166
13
votes
Accepted
Please review my sketch of the Machine Learning process
This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the ...
- 4,103
10
votes
Image resizing and padding for CNN
You have a few options:
For Small Images:
upsample through interpolation
pad the image using zeros
If you are unable to maintain the aspect ratio via upsampling, you can upsample and also crop the ...
- 2,522
9
votes
Accepted
What is a benchmark model?
Benchmarking is the process of comparing your result to existing methods. You may compare to published results using another paper, for example. If there is no other obvious methodology against which ...
- 3,077
9
votes
Accepted
How to get spike values from a value sequence?
This is very simple. Let's say your data in Panda format (named data_df), and extracting peaks/spikes over a certain threshold (e.g. 15000 here) is simply:
...
- 4,207
9
votes
Accepted
How to export PCA to use in another program
Ideally PCA should not be used as a part of pre-processing feature reduction.
Anyhow regarding saving and reusing PCA model, sharing a basic code snippet which is working very fine in my case(as I'm ...
- 1,252
8
votes
Accepted
How to implement global contrast normalization in python?
there are multiple issues with the code:
You force the values in the image to be uint8 (8-bit integer). Since the values are floats they will be casted/rounded to either 0 or 1.
This will later be ...
- 96
8
votes
Accepted
sklearn SimpleImputer too slow for categorical data represented as string values
Searching the source code of Sklearn for SimpleImputer (with strategy= "most_frequent"), the most frequent value is calculated within a loop in python, therefore that is the part of code that is so ...
- 501
7
votes
Accepted
Data preprocessing: Should we normalise images pixel-wise?
We first center our data by subtracting the mean of the batch. We also divide by the standard deviation, so our formula becomes:
$ z = \frac{x - \mu}{\sigma} $
where:
$ x $ is the pixel value
$ \...
- 2,522
7
votes
Different Test Set and Training Set Distribution
I want to subsample observations from training set which closely
resembles test set.
I am not sure you'd want to do that. The whole purpose is rather to train your algorithm so that it generalises ...
- 201
7
votes
StandardScaler before or after splitting data - which is better?
You shouldn't be doing fit_transform(X_test) on the test data.
The fit already occurred above.
...
- 71
7
votes
Accepted
A single column has many values per row, separated by a comma. How to create an individual column for each of these?
It would be better if you could provide some code which allows us to reproduce at least part of your DataFrame, such as this:
...
- 763
7
votes
Loading own train data and labels in dataloader using pytorch?
I think the standard way is to create a Dataset class object from the arrays and pass the Dataset object to the ...
- 171
7
votes
Accepted
Using pandas get_dummies() on real world unseen data
Yes, the encoding would be lost. You should instead use sklearn OneHotEncoder and save the corresponding encoder instance so that you can re-load it on unseen data.
...
- 566
7
votes
Why is oversampling outperforming class weight?
It could be your sampling strategy
If you are oversampling by just duplicating data from class 0, then it is likely that you are overfitting. The same datapoint ...
- 3,518
7
votes
Out of range [0,1] MinMaxScaler for test data
From documentation:
MinMaxScaler
X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0))
X_scaled = X_std * (max - min) + min
...
- 396
6
votes
Accepted
Dealing with training set of questionable quality
Welcome to the real world of data science. Here, the data sets are not as clean as you thought while doing those courses/tutorials online. Those are super polished and refined. But, the real world ...
- 8,256
6
votes
One Hot Encoding vs Word Embedding - When to choose one or another?
It seems that Embedding vector is the best solution here.
However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick".
In this variant, when the number of unique words ...
- 296
6
votes
Accepted
How distribution of data effects model performance?
Good question. Your interpretation is adequate. Using a logarithmic function reduces the skewness of the target variable. Why does that matter?
Transforming your target via a logarithmic function ...
- 5,446
6
votes
Accepted
Effect of Stop-Word Removal on Transformers for Text Classification
Very interesting question.
Easy, but probably lazy answer
When using pre-trained models, it is always advised to feed it data similar to what it was trained with. Basically, if it matters, don't ...
- 5,446
Only top scored, non community-wiki answers of a minimum length are eligible