Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
41
votes
2answers
40k views

How to prepare/augment images for neural network?

I would like to use a neural network for image classification. I'll start with pre-trained CaffeNet and train it for my application. How should I prepare the input images? In this case, all the ...
33
votes
3answers
35k views

StandardScaler before and after splitting data

When I was reading about using StandardScaler, most of the recommendations were saying that you should use StandardScaler before ...
30
votes
3answers
16k views

Difference between OrdinalEncoder and LabelEncoder

I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing: In the Documentation it is given about ...
19
votes
4answers
18k views

Different Test Set and Training Set Distribution

I am working on a data science competition for which the distribution of my test set is different from the training set. I want to subsample observations from training set which closely resembles test ...
16
votes
3answers
30k views

Image resizing and padding for CNN

I want to train a CNN for image recognition. Images for training have not fixed size. I want the input size for the CNN to be 50x100 (height x width), for example. When I resize some small sized ...
13
votes
2answers
21k views

Loading own train data and labels in dataloader using pytorch?

I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader? I have a dataset that I created and the ...
12
votes
2answers
16k views

One Hot Encoding vs Word Embedding - When to choose one or another?

A colleague of mine is having an interesting situation, he has quite a large set of possibilities for a defined categorical feature (+/- 300 different values) The usual data science approach would be ...
10
votes
5answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
9
votes
1answer
7k views

Data preprocessing: Should we normalise images pixel-wise?

Let me present you with a toy example and a reasoning on image normalisation I had: Suppose we have a CNN architecture to classify NxN grayscale images in two categories. Pixel values range from 0 (...
9
votes
1answer
4k views

How to approach the numer.ai competition with anonymous scaled numerical predictors?

Numer.ai has been around for a while now and there seem to be only few posts or other discussions about it on the web. The system has changed from time to time and the set-up today is the following: ...
8
votes
1answer
4k views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a Word2Vec or Glove,...
7
votes
1answer
10k views

How to preprocess different kinds of data (continuous, discrete, categorical) before Decision Tree learning

I want to use some Decision Tree learning, such as the Random Forest classifier. I have data of different types: continuous, discrete and categorical. How do I have to preprocess data in order to ...
7
votes
1answer
7k views

How to implement global contrast normalization in python?

I am trying to implement global contrast normalization in python from Yoshua Bengio's Deep Learning book (section 12.2.1.1 pg. 442). From the book, to get a normalized image using global contrast ...
7
votes
1answer
2k views

sklearn SimpleImputer too slow for categorical data represented as string values

I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s SimpleImputer but ...
7
votes
1answer
2k views

Extracting individual emails from an email thread

Most of the open source datasets are well formatted i.e each email message is separated well like the enron email dataset. But out in the real world it is highly difficult to separate a top email ...
6
votes
7answers
5k views

How to define a distance measure between two IP addresses?

I have IP addresses as feature and I would like to know how much two IP addresses are similar to each other to use the difference in an Euclidean distance measure (in order to quantify the ...
6
votes
5answers
19k views

issue with oneHotEncoding

So i have a PandasDataFrame with categorical variables in a column which i want to one hot encode i've used the following code from an ML udemy course ...
6
votes
2answers
202 views

Dealing with training set of questionable quality

Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into ...
6
votes
2answers
260 views

How distribution of data effects model performance?

I am working on House Prices: Advanced Regression Techniques dataset. I was going through some kernels noticed many people converted SalePrice to ...
6
votes
2answers
11k views

Why is input preprocessing in VGG16 in Keras not 1/255.0

I am just trying to use pre-trained vgg16 to make prediction in Keras like this. ...
6
votes
2answers
515 views

Best way to scale across different datasets

I have come across a peculiar situation when preprocessing data. Let's say I have a dataset A. I split the dataset into A_train ...
5
votes
3answers
13k views

How to get spike values from a value sequence?

I have pile of vectors where the values could be plotted like this: Now I want to extract the "spike values" (over a certain threshold say 15,000). In this case there is fifteen. How could this be ...
5
votes
1answer
10k views

What is a benchmark model?

I am working on a breast cancer dataset (http://kdd.org/kdd-cup/view/kdd-cup-2008). I need to perform classification on the data using C4.5 algorithm, after doing any necessary pre-processing. A ...
5
votes
1answer
654 views

Should I standardize first or generate polynomials first?

Recently I am dealing a classification problem with some algorithms, say logistic regression. When I preprocess my data, I standardize all my features and then generate polynomial features based on ...
5
votes
3answers
541 views

Do you apply outlier detection of numerical data in practical applications?

In data science we often get raw data to work on. It is the main task to draw conclusions from the training data that can be generalized to future unseen data. Do you apply outlier detection in your ...
5
votes
2answers
2k views

Columntransformer multiple columns with vector inputs

This is perhaps more of a coding question than data science so apologies if this is not the right platform to ask this. My question is related to the sklearn's <...
4
votes
3answers
2k views

How would one separate digits for number recognition?

Assuming that one has a neural network capable of returning the numerical digit from a given image of size 28x28px. How would one split an image of unknown size and an unknown amount of digits into a ...
4
votes
2answers
5k views

Convert exponential to normal distribution

For the distribution shown below, I want to convert the exponential distribution to a normal distribution. I want to do this is as part of data pre-processing so that the classifier can better ...
4
votes
2answers
110 views

If there are no missing values in our training set, should we accommodate missing values in an unseen test set?

My training data has no missing values. I'm unsure whether or not I should fit say, imputation, on the training set so that I can accommodate possible missing values on the test set, because the test ...
4
votes
2answers
2k views

Nested features with one to many relationships

I have a dataset that has (among others) a categorical variable with many levels and further attributes associated with each level. For example, consider predicting machine failure based on its last ...
4
votes
3answers
289 views

Reducing the size of a dataset

I am trying to classify gestures. I am using Python's scikit learn library classification algorithms for that. I have collected depth images for this purpose. 200 samples are collected for each ...
4
votes
2answers
412 views

How to choose best classifier for Low positive to negative class ratio in data (training, validation and real time)?

Positive class is ~4%. The class weight methods will not work as even if I balance the data while training by scaling positive class samples, in real time (or in test data), the distribution is ...
4
votes
1answer
25 views

How to perform data scaling/standardization on dataset containing grouped values?

So I have a dataset containing the results of executing problem instances with different given solver strategies. Simplified example: ...
4
votes
1answer
165 views

Feature importance over a subset of instance space instead of an entire instance space

I'm really curious if anyone has faced this problem before, or is it even widely studied at all. Imagine we have a feature that isn't important (based on many widely available and textbook feature ...
4
votes
2answers
145 views

Normalising data with multiple methods

When training a neural network, I appreciate that data normalisation helps training. However, is it a good idea to normalise the data in multiple ways. For instance, is it a good idea to apply z-score ...
4
votes
1answer
3k views

One hot encoding of target space

I had a face to face interview for a data scientist job a few days ago. One of the questions I was asked was: in the case of classifier predicting the brand of TV from some features (price, size, ...
3
votes
2answers
960 views

Is it acceptable not to transform() test data after train data is being fit_transform()-ed

We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, ...
3
votes
1answer
2k views

Using pandas get_dummies() on real world unseen data

I made a ML model, trained and tested it with my data containing categorical variables. To create dummy variables I used pd.get_dummies() before the split. I now ...
3
votes
2answers
4k views

A single column has many values per row, separated by a comma. How to create an individual column for each of these?

As you can see below, I have a column called code with multiple values per row, separated by a comma. How can I create a column for each of these codes and make ...
3
votes
1answer
301 views

Why is oversampling outperforming class weight?

I have a dataset that is highly imbalanced. One class has 412 (class 0) samples while the other has 67215 (class 1) samples. For its classification, I am using MLP. When I use class weight of 165 for ...
3
votes
2answers
5k views

How to export PCA to use in another program

I'm trying to write a random forest classifier for a very large dataset, as such as part of the pre-processing i have applied PCA to reduce from 643 features to 5 PC's. Is it possible to export these ...
3
votes
5answers
4k views

Large no of categorical variables with large no of categories

I'm working on a binary classification problem where the dataset is slightly imbalanced (30% class 0 | 70% class 1). Most of my features are categorical with large number of categories. For example: ...
3
votes
1answer
2k views

SVM SMOTE fit_resample() function runs forever with no result

Problem fit_resample(X,y) is taking too long to complete execution for 2million rows. Dataset specifications I have a labeled dataset about network features, ...
3
votes
1answer
2k views

Denoising Autoenoders with Variable Length Input

I'm working on a problem with data from a continuous real-valued signal. The goal is to use ML to smooth the signal based off of past data. To accomplish this, the signal is windowed into a period ...
3
votes
1answer
52 views

How data are prepared during training, testing and in production?

Most of real world datasets have features with missing values. Replacing missing values with an appropriate value such as its mean, is considered as a good step in feature engineering. Some times we ...
3
votes
2answers
129 views

Repeated features in Neural Networks with tabular data

When using algorithms like linear regression or least-squares methods, having repeated or highly correlated features can be harmful for the model. For tree based models, they are generally not too ...
3
votes
1answer
67 views

What is the proper order of normalization steps before and after splitting data

I use a classification model on time-series data where I normalize the data before splitting the data into train and test. Now, I know that train and test data should be treated separately to prevent ...
3
votes
1answer
4k views

Iterate over multiple dataframe rows at the same time

I have 16 different dataframes with the same number of rows/columns and another 2 separate dataframes with that same shape that i'm using to compare with the 16 dataframe values. I need to loop over ...
3
votes
2answers
83 views

How can I perform categorical encoding when the dataset is too large for memory?

I generally do preprocessing before fitting estimators using Scikit-Learn. My latest project is using significantly more data than I have used in the past, and whilst I know I can use online learning ...
3
votes
1answer
116 views

ASR on low dataset

I am doing an ASR(automatic speech recognition) as master thesis on low key dataset. Voice and text data is labelled. There are around 4000 phrases and around 5 hours speech. I don't have background ...

1
2 3 4 5
8