Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
40
votes
2answers
39k views

How to prepare/augment images for neural network?

I would like to use a neural network for image classification. I'll start with pre-trained CaffeNet and train it for my application. How should I prepare the input images? In this case, all the ...
24
votes
3answers
21k views

StandardScaler before and after splitting data

When I was reading about using StandardScaler, most of the recommendations were saying that you should use StandardScaler before ...
21
votes
3answers
8k views

Difference between OrdinalEncoder and LabelEncoder

I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing: In the Documentation it is given about ...
16
votes
4answers
14k views

Different Test Set and Training Set Distribution

I am working on a data science competition for which the distribution of my test set is different from the training set. I want to subsample observations from training set which closely resembles test ...
15
votes
3answers
21k views

Image resizing and padding for CNN

I want to train a CNN for image recognition. Images for training have not fixed size. I want the input size for the CNN to be 50x100 (height x width), for example. When I resize some small sized ...
10
votes
5answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
9
votes
1answer
3k views

How to approach the numer.ai competition with anonymous scaled numerical predictors?

Numer.ai has been around for a while now and there seem to be only few posts or other discussions about it on the web. The system has changed from time to time and the set-up today is the following: ...
8
votes
1answer
6k views

Data preprocessing: Should we normalise images pixel-wise?

Let me present you with a toy example and a reasoning on image normalisation I had: Suppose we have a CNN architecture to classify NxN grayscale images in two categories. Pixel values range from 0 (...
7
votes
1answer
10k views

How to preprocess different kinds of data (continuous, discrete, categorical) before Decision Tree learning

I want to use some Decision Tree learning, such as the Random Forest classifier. I have data of different types: continuous, discrete and categorical. How do I have to preprocess data in order to ...
7
votes
2answers
11k views

One Hot Encoding vs Word Embeding - When to choose one or another?

A colleague of mine is having an interesting situation, he has quite a large set of possibilities for a defined categorical feature (+/- 300 different values) The usual data science approach would be ...
7
votes
1answer
2k views

Extracting individual emails from an email thread

Most of the open source datasets are well formatted i.e each email message is separated well like the enron email dataset. But out in the real world it is highly difficult to separate a top email ...
6
votes
7answers
4k views

How to define a distance measure between two IP addresses?

I have IP addresses as feature and I would like to know how much two IP addresses are similar to each other to use the difference in an Euclidean distance measure (in order to quantify the ...
6
votes
2answers
197 views

Dealing with training set of questionable quality

Most of the material I have read in the past usually assumes that the training set is flawless. However that doesn't seem to be the case here with what I am given. The data that is meant to send into ...
6
votes
2answers
193 views

How distribution of data effects model performance?

I am working on House Prices: Advanced Regression Techniques dataset. I was going through some kernels noticed many people converted SalePrice to ...
6
votes
1answer
6k views

How to implement global contrast normalization in python?

I am trying to implement global contrast normalization in python from Yoshua Bengio's Deep Learning book (section 12.2.1.1 pg. 442). From the book, to get a normalized image using global contrast ...
6
votes
2answers
12k views

Loading own train data and labels in dataloader using pytorch?

I have x_data and labels separately. How can I combine and load them in the model using torch.utils.data.DataLoader? I have a dataset that I created and the ...
6
votes
1answer
1k views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a W2V or Glove, ...
6
votes
2answers
295 views

Best way to scale across different datasets

I have come across a peculiar situation when preprocessing data. Let's say I have a dataset A. I split the dataset into A_train ...
5
votes
6answers
15k views

issue with oneHotEncoding

So i have a PandasDataFrame with categorical variables in a column which i want to one hot encode i've used the following code from an ML udemy course ...
5
votes
2answers
8k views

Why is input preprocessing in VGG16 in Keras not 1/255.0

I am just trying to use pre-trained vgg16 to make prediction in Keras like this. ...
5
votes
3answers
508 views

Do you apply outlier detection of numerical data in practical applications?

In data science we often get raw data to work on. It is the main task to draw conclusions from the training data that can be generalized to future unseen data. Do you apply outlier detection in your ...
5
votes
2answers
1k views

Columntransformer multiple columns with vector inputs

This is perhaps more of a coding question than data science so apologies if this is not the right platform to ask this. My question is related to the sklearn's <...
5
votes
2answers
2k views

Nested features with one to many relationships

I have a dataset that has (among others) a categorical variable with many levels and further attributes associated with each level. For example, consider predicting machine failure based on its last ...
4
votes
3answers
11k views

How to get spike values from a value sequence?

I have pile of vectors where the values could be plotted like this: Now I want to extract the "spike values" (over a certain threshold say 15,000). In this case there is fifteen. How could this be ...
4
votes
1answer
8k views

What is a benchmark model?

I am working on a breast cancer dataset (http://kdd.org/kdd-cup/view/kdd-cup-2008). I need to perform classification on the data using C4.5 algorithm, after doing any necessary pre-processing. A ...
4
votes
3answers
1k views

How would one separate digits for number recognition?

Assuming that one has a neural network capable of returning the numerical digit from a given image of size 28x28px. How would one split an image of unknown size and an unknown amount of digits into a ...
4
votes
1answer
419 views

Should I standardize first or generate polynomials first?

Recently I am dealing a classification problem with some algorithms, say logistic regression. When I preprocess my data, I standardize all my features and then generate polynomial features based on ...
4
votes
2answers
392 views

How to choose best classifier for Low positive to negative class ratio in data (training, validation and real time)?

Positive class is ~4%. The class weight methods will not work as even if I balance the data while training by scaling positive class samples, in real time (or in test data), the distribution is ...
4
votes
1answer
120 views

Feature importance over a subset of instance space instead of an entire instance space

I'm really curious if anyone has faced this problem before, or is it even widely studied at all. Imagine we have a feature that isn't important (based on many widely available and textbook feature ...
4
votes
2answers
106 views

Normalising data with multiple methods

When training a neural network, I appreciate that data normalisation helps training. However, is it a good idea to normalise the data in multiple ways. For instance, is it a good idea to apply z-score ...
3
votes
2answers
192 views

Is it acceptable not to transform() test data after train data is being fit_transform()-ed

We know that the best practice in data preprocessing (such as standardization, Normalization, ... etc) is that while we perform fit_trasform() on the training data, ...
3
votes
2answers
2k views

A single column has many values per row, separated by a comma. How to create an individual column for each of these?

As you can see below, I have a column called code with multiple values per row, separated by a comma. How can I create a column for each of these codes and make ...
3
votes
1answer
63 views

Why is oversampling outperforming class weight?

I have a dataset that is highly imbalanced. One class has 412 (class 0) samples while the other has 67215 (class 1) samples. For its classification, I am using MLP. When I use class weight of 165 for ...
3
votes
1answer
625 views

sklearn SimpleImputer too slow for categorical data represented as string values

I have a data set with categorical features represented as string values and I want to fill-in missing values in it. I’ve tried to use sklearn’s SimpleImputer but ...
3
votes
1answer
1k views

SVM SMOTE fit_resample() function runs forever with no result

Problem fit_resample(X,y) is taking too long to complete execution for 2million rows. Dataset specifications I have a labeled dataset about network features, ...
3
votes
1answer
2k views

Denoising Autoenoders with Variable Length Input

I'm working on a problem with data from a continuous real-valued signal. The goal is to use ML to smooth the signal based off of past data. To accomplish this, the signal is windowed into a period ...
3
votes
1answer
27 views

What is the proper order of normalization steps before and after splitting data

I use a classification model on time-series data where I normalize the data before splitting the data into train and test. Now, I know that train and test data should be treated separately to prevent ...
3
votes
1answer
588 views

Iterate over multiple dataframe rows at the same time

I have 16 different dataframes with the same number of rows/columns and another 2 separate dataframes with that same shape that i'm using to compare with the 16 dataframe values. I need to loop over ...
3
votes
1answer
90 views

ASR on low dataset

I am doing an ASR(automatic speech recognition) as master thesis on low key dataset. Voice and text data is labelled. There are around 4000 phrases and around 5 hours speech. I don't have background ...
3
votes
2answers
44 views

how to handle values that only appear once in a column?

Counting the values of a column using pandas I got the following result: ...
3
votes
1answer
2k views

Why does not log transformation make the data normalized?

Having some skewed features as shown in the following figure. I am trying to imply log transformation to the feature called vBMD(mgHA/cm3). I run the following ...
3
votes
2answers
3k views

How to scale prediction back after preprocessing

So I'm a newbie to machine learning and am currently using the iris data set. I ran through a quick online tutorial about predicting stock prices and thought I'd try and do the iris one myself. The ...
3
votes
2answers
50 views

How can I perform categorical encoding when the dataset is too large for memory?

I generally do preprocessing before fitting estimators using Scikit-Learn. My latest project is using significantly more data than I have used in the past, and whilst I know I can use online learning ...
3
votes
1answer
348 views

Normalizing test data

I have a problem in data normalization. I have data for which I need to create an SVM. I will be using the model for real-time predictions. I know that the test tuples should be normalized using the ...
3
votes
2answers
1k views

Machine Learning or Survival Analysis?

I am working on building prediction model for disk failures (time taken to occur a disk failure and what parameters could strongly affect disk failures). I am bit confused on- What data preprocessing ...
3
votes
2answers
390 views

Sampling for multi categorical variable

My hypothesis h depends on multiple categorical variables (a,b,c) each with their corresponding set of possible values ...
3
votes
1answer
94 views

binning high cardinality categorical features

one approach I have tried when preprocessing high cardinality categorical features (for example, US City) is to do a value count of all the values in the data, then take the top x most frequently ...
3
votes
1answer
261 views

Standard correlation coefficient of various datasets

I understood the correlation coefficient of the first line. But the correlation coefficient of second and third line differ with the first line in figure 1. Why is it so ?. Even the shapes differ for ...
3
votes
1answer
170 views

Video classification of birds

I'm working on a project where the task is to classify videos of birds and predict the species. I have tried to use machine learning models originally designed for deep action recognition and tweaked ...
3
votes
1answer
3k views

One hot encoding of target space

I had a face to face interview for a data scientist job a few days ago. One of the questions I was asked was: in the case of classifier predicting the brand of TV from some features (price, size, ...

1
2 3 4 5
7