# Tag Info

Accepted

### StandardScaler before or after splitting data - which is better?

In the interest of preventing information about the distribution of the test set leaking into your model, you should go for option #2 and fit the scaler on your training data only, then standardise ...
• 1,578

### Difference between OrdinalEncoder and LabelEncoder

Afaik, both have the same functionality. A bit difference is the idea behind. OrdinalEncoder is for converting features, while ...
• 756
Accepted

### How to prepare/augment images for neural network?

The idea with Neural Networks is that they need little pre-processing since the heavy lifting is done by the algorithm which is the one in charge of learning the features. The winners of the Data ...
• 3,290

### Different Test Set and Training Set Distribution

Great question, this is what is known in Machine Learning paradigm as either "Covariate Shift", or "Model Drift" or "Nonstationarity" and so on. One of the critical assumption one would make to ...
• 4,046
Accepted

### One Hot Encoding vs Word Embedding - When to choose one or another?

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each ...
• 17.4k

### Difference between OrdinalEncoder and LabelEncoder

As for differences in OrdinalEncoder and LabelEncoder implementation, the accepted answer mentions the shape of the data: ...
• 341
Accepted

### Image resizing and padding for CNN

This question on stackoverflow might help you. To sum up, some deep learning researchers think that padding a big part of the image is not a good practice, since the neural network has to learn that ...
• 5,714
Accepted

### In ML why selecting the best variables?

You are right. If someone is using regularization correctly and doing hyperparameter tuning to avoid overfitting, then it should not be a problem theoretically (ie multi-collinearity will not reduce ...

Assuming both of x_data and labels are lists or numpy arrays, ...
• 250
Accepted

### How to preprocess different kinds of data (continuous, discrete, categorical) before Decision Tree learning

One of the benefits of decision trees is that ordinal (continuous or discrete) input data does not require any significant preprocessing. In fact, the results should be consistent regardless of any ...
Accepted

### Please review my sketch of the Machine Learning process

This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the ...
• 3,898

### Encoding with OrdinalEncoder : how to give levels as user input?

I'm not sure if you ever figured this out but I was trying to find answers on this exact same question and there aren't really any good answers in my opinion. I finally figured it out though. ...
• 121
Accepted

### What is a benchmark model?

Benchmarking is the process of comparing your result to existing methods. You may compare to published results using another paper, for example. If there is no other obvious methodology against which ...
• 3,037

### Image resizing and padding for CNN

You have a few options: For Small Images: upsample through interpolation pad the image using zeros If you are unable to maintain the aspect ratio via upsampling, you can upsample and also crop the ...
• 2,462
Accepted

### How to implement global contrast normalization in python?

there are multiple issues with the code: You force the values in the image to be uint8 (8-bit integer). Since the values are floats they will be casted/rounded to either 0 or 1. This will later be ...
• 96
Accepted

### How to get spike values from a value sequence?

This is very simple. Let's say your data in Panda format (named data_df), and extracting peaks/spikes over a certain threshold (e.g. 15000 here) is simply: ...
• 4,046
Accepted

### sklearn SimpleImputer too slow for categorical data represented as string values

Searching the source code of Sklearn for SimpleImputer (with strategy= "most_frequent"), the most frequent value is calculated within a loop in python, therefore that is the part of code that is so ...
Accepted

### Data preprocessing: Should we normalise images pixel-wise?

We first center our data by subtracting the mean of the batch. We also divide by the standard deviation, so our formula becomes: $z = \frac{x - \mu}{\sigma}$ where: $x$ is the pixel value \$ \...
• 2,462

### Different Test Set and Training Set Distribution

I want to subsample observations from training set which closely resembles test set. I am not sure you'd want to do that. The whole purpose is rather to train your algorithm so that it generalises ...
• 201

### One Hot Encoding vs Word Embedding - When to choose one or another?

It seems that Embedding vector is the best solution here. However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick". In this variant, when the number of unique words ...
• 306
Accepted

### A single column has many values per row, separated by a comma. How to create an individual column for each of these?

It would be better if you could provide some code which allows us to reproduce at least part of your DataFrame, such as this: ...

I think the standard way is to create a Dataset class object from the arrays and pass the Dataset object to the ...
• 171
Accepted

### Using pandas get_dummies() on real world unseen data

Yes, the encoding would be lost. You should instead use sklearn OneHotEncoder and save the corresponding encoder instance so that you can re-load it on unseen data. ...
• 546
Accepted

### How to export PCA to use in another program

Ideally PCA should not be used as a part of pre-processing feature reduction. Anyhow regarding saving and reusing PCA model, sharing a basic code snippet which is working very fine in my case(as I'm ...
• 1,224

### Why is oversampling outperforming class weight?

It could be your sampling strategy If you are oversampling by just duplicating data from class 0, then it is likely that you are overfitting. The same datapoint ...
• 3,338
Accepted

### Dealing with training set of questionable quality

Welcome to the real world of data science. Here, the data sets are not as clean as you thought while doing those courses/tutorials online. Those are super polished and refined. But, the real world ...
• 8,076
Accepted

### How distribution of data effects model performance?

Good question. Your interpretation is adequate. Using a logarithmic function reduces the skewness of the target variable. Why does that matter? Transforming your target via a logarithmic function ...
• 5,206
Accepted

### How to define a distance measure between two IP addresses?

If I understood them correctly, both Jeremy and Edmund's (first) solutions are the same, namely, plain euclidean distance in a 4-dimensional space of IP addresses.BTW, I think a very fast alternative ...
• 481