49 votes

Difference between OrdinalEncoder and LabelEncoder

Afaik, both have the same functionality. A bit difference is the idea behind. OrdinalEncoder is for converting features, while ...
ipramusinto's user avatar
47 votes
Accepted

StandardScaler before or after splitting data - which is better?

In the interest of preventing information about the distribution of the test set leaking into your model, you should go for option #2 and fit the scaler on your training data only, then standardise ...
redhqs's user avatar
  • 1,678
28 votes
Accepted

One Hot Encoding vs Word Embedding - When to choose one or another?

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each ...
Brian Spiering's user avatar
28 votes

Difference between OrdinalEncoder and LabelEncoder

As for differences in OrdinalEncoder and LabelEncoder implementation, the accepted answer mentions the shape of the data: ...
Nate Anderson's user avatar
25 votes

Different Test Set and Training Set Distribution

Great question, this is what is known in Machine Learning paradigm as either "Covariate Shift", or "Model Drift" or "Nonstationarity" and so on. One of the critical assumption one would make to ...
TwinPenguins's user avatar
  • 4,249
18 votes
Accepted

Image resizing and padding for CNN

This question on stackoverflow might help you. To sum up, some deep learning researchers think that padding a big part of the image is not a good practice, since the neural network has to learn that ...
David Masip's user avatar
  • 6,051
18 votes
Accepted

In ML why selecting the best variables?

You are right. If someone is using regularization correctly and doing hyperparameter tuning to avoid overfitting, then it should not be a problem theoretically (ie multi-collinearity will not reduce ...
fractalnature's user avatar
16 votes

Loading own train data and labels in dataloader using pytorch?

Assuming both of x_data and labels are lists or numpy arrays, ...
ASHu2's user avatar
  • 260
16 votes
Accepted

Encoding with OrdinalEncoder : how to give levels as user input?

I'm not sure if you ever figured this out but I was trying to find answers on this exact same question and there aren't really any good answers in my opinion. I finally figured it out though. ...
fugumagu's user avatar
  • 176
13 votes
Accepted

Please review my sketch of the Machine Learning process

This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the ...
Simon Larsson's user avatar
10 votes

Image resizing and padding for CNN

You have a few options: For Small Images: upsample through interpolation pad the image using zeros If you are unable to maintain the aspect ratio via upsampling, you can upsample and also crop the ...
Ben's user avatar
  • 2,562
9 votes
Accepted

How to get spike values from a value sequence?

This is very simple. Let's say your data in Panda format (named data_df), and extracting peaks/spikes over a certain threshold (e.g. 15000 here) is simply: ...
TwinPenguins's user avatar
  • 4,249
9 votes
Accepted

How to export PCA to use in another program

Ideally PCA should not be used as a part of pre-processing feature reduction. Anyhow regarding saving and reusing PCA model, sharing a basic code snippet which is working very fine in my case(as I'm ...
vipin bansal's user avatar
  • 1,262
8 votes
Accepted

How to implement global contrast normalization in python?

there are multiple issues with the code: You force the values in the image to be uint8 (8-bit integer). Since the values are floats they will be casted/rounded to either 0 or 1. This will later be ...
someone's user avatar
  • 96
8 votes

StandardScaler before or after splitting data - which is better?

You shouldn't be doing fit_transform(X_test) on the test data. The fit already occurred above. ...
starsini's user avatar
8 votes
Accepted

sklearn SimpleImputer too slow for categorical data represented as string values

Searching the source code of Sklearn for SimpleImputer (with strategy= "most_frequent"), the most frequent value is calculated within a loop in python, therefore that is the part of code that is so ...
Giannis Krilis's user avatar
7 votes
Accepted

Data preprocessing: Should we normalise images pixel-wise?

We first center our data by subtracting the mean of the batch. We also divide by the standard deviation, so our formula becomes: $ z = \frac{x - \mu}{\sigma} $ where: $ x $ is the pixel value $ \...
Ben's user avatar
  • 2,562
7 votes

Different Test Set and Training Set Distribution

I want to subsample observations from training set which closely resembles test set. I am not sure you'd want to do that. The whole purpose is rather to train your algorithm so that it generalises ...
arnaud's user avatar
  • 201
7 votes
Accepted

A single column has many values per row, separated by a comma. How to create an individual column for each of these?

It would be better if you could provide some code which allows us to reproduce at least part of your DataFrame, such as this: ...
ignoring_gravity's user avatar
7 votes

Loading own train data and labels in dataloader using pytorch?

I think the standard way is to create a Dataset class object from the arrays and pass the Dataset object to the ...
Johannes's user avatar
  • 171
7 votes
Accepted

Using pandas get_dummies() on real world unseen data

Yes, the encoding would be lost. You should instead use sklearn OneHotEncoder and save the corresponding encoder instance so that you can re-load it on unseen data. ...
gented's user avatar
  • 566
7 votes

Why is oversampling outperforming class weight?

It could be your sampling strategy If you are oversampling by just duplicating data from class 0, then it is likely that you are overfitting. The same datapoint ...
Bruno Lubascher's user avatar
7 votes

Out of range [0,1] MinMaxScaler for test data

From documentation: MinMaxScaler X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0)) X_scaled = X_std * (max - min) + min ...
Andrew's user avatar
  • 406
6 votes

One Hot Encoding vs Word Embedding - When to choose one or another?

It seems that Embedding vector is the best solution here. However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick". In this variant, when the number of unique words ...
michaelg's user avatar
  • 296
6 votes

Preprocessing for Text Classification in Transformer Models (BERT variants)

A quick experiment you can do is to once do the preprocessing steps that you usually do and then feed it to the model and get the results. And once feed the dataset as it is to the model to compare ...
Fatemeh Rahimi's user avatar
6 votes
Accepted

How distribution of data effects model performance?

Good question. Your interpretation is adequate. Using a logarithmic function reduces the skewness of the target variable. Why does that matter? Transforming your target via a logarithmic function ...
Valentin Calomme's user avatar
6 votes
Accepted

Is it acceptable not to transform() test data after train data is being fit_transform()-ed

No, it does not make sense to do this. You model has learned how to map one input space to another, that is to say it is itself function approximation, and will likely not know what to for the unseen ...
n1k31t4's user avatar
  • 14.9k
6 votes
Accepted

Effect of Stop-Word Removal on Transformers for Text Classification

Very interesting question. Easy, but probably lazy answer When using pre-trained models, it is always advised to feed it data similar to what it was trained with. Basically, if it matters, don't ...
Valentin Calomme's user avatar
5 votes

Real time noise removal using Savitzky-Golay Method

The original Savitzky-Golay paper addressed smoothing, meaning that you estimate a value for some point in the past, using values from its past and future. It sounds like what you're interested in is ...
gms's user avatar
  • 171
5 votes

issue with oneHotEncoding

There is an easy way to use one hot encoding in pandas and you can read about it in the following link: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.get_dummies.html The ...
AR_'s user avatar
  • 91

Only top scored, non community-wiki answers of a minimum length are eligible