45 votes
Accepted

StandardScaler before or after splitting data - which is better?

In the interest of preventing information about the distribution of the test set leaking into your model, you should go for option #2 and fit the scaler on your training data only, then standardise ...
  • 1,578
42 votes

Difference between OrdinalEncoder and LabelEncoder

Afaik, both have the same functionality. A bit difference is the idea behind. OrdinalEncoder is for converting features, while ...
35 votes
Accepted

How to prepare/augment images for neural network?

The idea with Neural Networks is that they need little pre-processing since the heavy lifting is done by the algorithm which is the one in charge of learning the features. The winners of the Data ...
  • 3,290
24 votes

Different Test Set and Training Set Distribution

Great question, this is what is known in Machine Learning paradigm as either "Covariate Shift", or "Model Drift" or "Nonstationarity" and so on. One of the critical assumption one would make to ...
  • 4,046
23 votes
Accepted

One Hot Encoding vs Word Embedding - When to choose one or another?

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each ...
22 votes

Difference between OrdinalEncoder and LabelEncoder

As for differences in OrdinalEncoder and LabelEncoder implementation, the accepted answer mentions the shape of the data: ...
17 votes
Accepted

Image resizing and padding for CNN

This question on stackoverflow might help you. To sum up, some deep learning researchers think that padding a big part of the image is not a good practice, since the neural network has to learn that ...
  • 5,714
17 votes
Accepted

In ML why selecting the best variables?

You are right. If someone is using regularization correctly and doing hyperparameter tuning to avoid overfitting, then it should not be a problem theoretically (ie multi-collinearity will not reduce ...
15 votes

Loading own train data and labels in dataloader using pytorch?

Assuming both of x_data and labels are lists or numpy arrays, ...
  • 250
13 votes
Accepted

How to preprocess different kinds of data (continuous, discrete, categorical) before Decision Tree learning

One of the benefits of decision trees is that ordinal (continuous or discrete) input data does not require any significant preprocessing. In fact, the results should be consistent regardless of any ...
13 votes
Accepted

Please review my sketch of the Machine Learning process

This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the ...
12 votes

Encoding with OrdinalEncoder : how to give levels as user input?

I'm not sure if you ever figured this out but I was trying to find answers on this exact same question and there aren't really any good answers in my opinion. I finally figured it out though. ...
  • 121
9 votes
Accepted

What is a benchmark model?

Benchmarking is the process of comparing your result to existing methods. You may compare to published results using another paper, for example. If there is no other obvious methodology against which ...
  • 3,037
9 votes

Image resizing and padding for CNN

You have a few options: For Small Images: upsample through interpolation pad the image using zeros If you are unable to maintain the aspect ratio via upsampling, you can upsample and also crop the ...
  • 2,462
8 votes
Accepted

How to implement global contrast normalization in python?

there are multiple issues with the code: You force the values in the image to be uint8 (8-bit integer). Since the values are floats they will be casted/rounded to either 0 or 1. This will later be ...
  • 96
8 votes
Accepted

How to get spike values from a value sequence?

This is very simple. Let's say your data in Panda format (named data_df), and extracting peaks/spikes over a certain threshold (e.g. 15000 here) is simply: ...
  • 4,046
8 votes
Accepted

sklearn SimpleImputer too slow for categorical data represented as string values

Searching the source code of Sklearn for SimpleImputer (with strategy= "most_frequent"), the most frequent value is calculated within a loop in python, therefore that is the part of code that is so ...
7 votes
Accepted

Data preprocessing: Should we normalise images pixel-wise?

We first center our data by subtracting the mean of the batch. We also divide by the standard deviation, so our formula becomes: $ z = \frac{x - \mu}{\sigma} $ where: $ x $ is the pixel value $ \...
  • 2,462
7 votes

Different Test Set and Training Set Distribution

I want to subsample observations from training set which closely resembles test set. I am not sure you'd want to do that. The whole purpose is rather to train your algorithm so that it generalises ...
  • 201
7 votes

One Hot Encoding vs Word Embedding - When to choose one or another?

It seems that Embedding vector is the best solution here. However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick". In this variant, when the number of unique words ...
  • 306
7 votes
Accepted

A single column has many values per row, separated by a comma. How to create an individual column for each of these?

It would be better if you could provide some code which allows us to reproduce at least part of your DataFrame, such as this: ...
7 votes

Loading own train data and labels in dataloader using pytorch?

I think the standard way is to create a Dataset class object from the arrays and pass the Dataset object to the ...
  • 171
7 votes
Accepted

Using pandas get_dummies() on real world unseen data

Yes, the encoding would be lost. You should instead use sklearn OneHotEncoder and save the corresponding encoder instance so that you can re-load it on unseen data. ...
  • 546
7 votes
Accepted

How to export PCA to use in another program

Ideally PCA should not be used as a part of pre-processing feature reduction. Anyhow regarding saving and reusing PCA model, sharing a basic code snippet which is working very fine in my case(as I'm ...
  • 1,224
7 votes

Why is oversampling outperforming class weight?

It could be your sampling strategy If you are oversampling by just duplicating data from class 0, then it is likely that you are overfitting. The same datapoint ...
6 votes
Accepted

Dealing with training set of questionable quality

Welcome to the real world of data science. Here, the data sets are not as clean as you thought while doing those courses/tutorials online. Those are super polished and refined. But, the real world ...
  • 8,076
6 votes
Accepted

How distribution of data effects model performance?

Good question. Your interpretation is adequate. Using a logarithmic function reduces the skewness of the target variable. Why does that matter? Transforming your target via a logarithmic function ...
5 votes
Accepted

How to define a distance measure between two IP addresses?

If I understood them correctly, both Jeremy and Edmund's (first) solutions are the same, namely, plain euclidean distance in a 4-dimensional space of IP addresses.BTW, I think a very fast alternative ...
  • 481
5 votes

Real time noise removal using Savitzky-Golay Method

The original Savitzky-Golay paper addressed smoothing, meaning that you estimate a value for some point in the past, using values from its past and future. It sounds like what you're interested in is ...
  • 171

Only top scored, non community-wiki answers of a minimum length are eligible