Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
33
votes
3answers
35k views

StandardScaler before and after splitting data

When I was reading about using StandardScaler, most of the recommendations were saying that you should use StandardScaler before ...
19
votes
4answers
18k views

Different Test Set and Training Set Distribution

I am working on a data science competition for which the distribution of my test set is different from the training set. I want to subsample observations from training set which closely resembles test ...
41
votes
2answers
40k views

How to prepare/augment images for neural network?

I would like to use a neural network for image classification. I'll start with pre-trained CaffeNet and train it for my application. How should I prepare the input images? In this case, all the ...
30
votes
3answers
16k views

Difference between OrdinalEncoder and LabelEncoder

I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing: In the Documentation it is given about ...
3
votes
1answer
52 views

How data are prepared during training, testing and in production?

Most of real world datasets have features with missing values. Replacing missing values with an appropriate value such as its mean, is considered as a good step in feature engineering. Some times we ...
8
votes
1answer
4k views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a Word2Vec or Glove,...
3
votes
2answers
129 views

Repeated features in Neural Networks with tabular data

When using algorithms like linear regression or least-squares methods, having repeated or highly correlated features can be harmful for the model. For tree based models, they are generally not too ...
3
votes
1answer
157 views

When to One-Hot encode categorical data when following Crisp-DM

I have a dataset that contains 15 categorical features (2 and 3 level factors which are non-ordinal) and 3 continuous numeric features. Seeing as most machine learning algorithms require numerical ...
9
votes
1answer
7k views

Data preprocessing: Should we normalise images pixel-wise?

Let me present you with a toy example and a reasoning on image normalisation I had: Suppose we have a CNN architecture to classify NxN grayscale images in two categories. Pixel values range from 0 (...
16
votes
3answers
30k views

Image resizing and padding for CNN

I want to train a CNN for image recognition. Images for training have not fixed size. I want the input size for the CNN to be 50x100 (height x width), for example. When I resize some small sized ...
3
votes
1answer
664 views

How to visualize data of a multidimensional dataset (TIMIT)

I've built a neural network for a speech recognition task using the timit dataset. I've extracted features using the perceptual linear prediction (PLP_ method. My features space has 39 dimensions (13 ...
3
votes
1answer
2k views

Using pandas get_dummies() on real world unseen data

I made a ML model, trained and tested it with my data containing categorical variables. To create dummy variables I used pd.get_dummies() before the split. I now ...
3
votes
1answer
262 views

Handling features with multiple values per instance in Python for Machine Learning model

I have a dataset which contains medical data about children and I am developing a predictive machine learning model to predict adverse pregnancy outcomes. The dataset contains mostly features with a ...
2
votes
3answers
1k views

Transformation of categorical variables (binary vs numerical)

When using categorical encoding, I see some authors use arbitrary numerical transformation while others use binary transformation. For example, if I have a feature vector with values A, B and c. The ...
2
votes
1answer
84 views

Image normalisation methods

I have found some research papers specifying explicitly the normalisation technique they used to get the results. What difference do IMG /255.0 And ...
0
votes
1answer
738 views

Data augmentation for multiple output heads in Keras

I have a transfer learning based two output classification problem. So, accordingly, I have formatted my data to have X_train as a ...
-1
votes
1answer
391 views

What are the best way to handle missing values [closed]

Suppose we have a dataframe df in python, with numerical and categorical variables. For Numerical, when do we replace by mean and when by median. For ...