Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

501 questions
Filter by
Sorted by
Tagged with
17 views

Encoding necessary for numerical data that can be summarised into a few groups

I have an input parameter that have 200 values. However, among the 200 values, there are only 3 distinct values. For example, like this: X1 10 14 14 10 22 22 10 10 14 . . . Should I treat this ...
21 views

Are There set in stones rules for EDA?

so I'm doing my first Data Science project and I'm having doubts in EDA and data preprocessing. There are several preprocess that I want to do, these are: dealing outliers dealing with missing ...
4 views

During preprocessing when I apply the func on few rows as the data is big it throws error of 'Series' object has no attribute 'lower'

def transform_text(Status_information): Status_information = Status_information.lower() #Status_information = Status_information.apply(lambda x: x.lower()) Status_information = nltk.word_tokenize(...
• 1
42 views

preprocess unbalanced skewed data

I am trying to find a way to preprocess my data. The data is as follow: study person_id energy_1 energy_2 y study_id A 2.3 -1.05 1 study_id2 B 1.03 0.04 0 Statistically speaking, we can see that ...
• 15
11 views

Dealing with categorical columns with unbalanced value count

I'm doing some data processing and wondering what is the best practice for dealing with categorical columns that has a value counts plot looking like the below (these are one-hot-encoded at a later ...
• 13
37 views

Predicting Buy, Sell, or Hold decisions classification

I have downloaded data from Yahoo Finance period: January 2010 to December 2019. I selected 20 different indexes representing some big name companies solely in the U.S. NASDAQ Market S&P 500. The ...
• 113
7 views

Data augmentation useful for this or not?

I have a training set of medical breast x-ray images. Approximately half of them are flipped along the horizontal axis since some are from the right side and some are from the left side. See the ...
46 views

Best way to encode product names in NLP?

I am supposed to train a classifier with historical shopping data that predicts the probability of an item being returned. The only human language contained for each purchase is the name of the ...
20 views

From what percentage should I remove a column that has empty values?

I have a problem. I am currently cleaning and preprocessing my data. Suppose I have the following data set ...
• 89
60 views

Should I split data into train/validation/test before feature scaling and feature selection or after?

I'm working on a project, I finished data preprocessing, and I found an article where it says that feature scaling and feature selection should be done after splitting data, some other articles say it ...
21 views

How to find a source?

I am given a dataset where I have to predict the distance. The training dataset consists of the values of the signal strength of 5 different sensors and the distance between the source and the 4 ...
• 121
27 views

At what point do I have to many dummy variables?

When does one-hot-encoding simply create too many dummy variables? For example, should I one hot encode a country name? This in the worst case could create close to 200 dummy variables. Should I one ...
15 views

Preserve relations between data points when preprocessing

I am tasked with a project that aims to predict the probability of a product being returned before the product is even ordered. I have an excel containing a bunch of orders. In order to make ...
1 vote
18 views

What is the best way for me to classify this audio data?

I have a set of audio data. I would like to classify each audio file based on a half-second of data from a give time period. The audio data is given as counts as a function of time $s(t)$. Right now ...
14 views

How do I match the number of the features of new text data to the data used in the training of the model

I am working on a classifier for some twitter data to predict who it was tweeted by. I am only using the text of the tweets to build the model. After all text related preprocessing here is how I ...
• 11
15 views

How to prepare data for multivariate prediction with irregular window size for prediction?

I have a dataset of different products and their possible configurations. I want to build a model which can predict the next part for the product given the previous part/parts. This model will be used ...
1 vote
57 views

Combining sklearn pipelines with different output shape

As part of a data preprocessing step, I'm trying to create a "master pipeline" from two separate pipelines, one for numerical features and one for datetime features. The numerical pipeline ...
• 13
116 views

Sklearn pipeline and custom transformers to remove specific value from columns

I'm trying to use sklearn pipelines and custom transformers to do outlier removal. What I want to do is identify outliers using an IQR-filter, set the outlier values to 'OUTLIER' (not NaN), and then ...
• 13
1 vote
24 views

Do the preprocessing steps for new data need to be identical to the steps for train/test data?

I'm using decision tree classification for a classification problem. I have preprocessed the data, train/test split it, and run a model with cross validation before testing it. The steps I followed ...
• 13
9 views

What is the procedure for data preprocessing for time-dependent LSTM classifier?

I attempt a beginner level LSTM classification task with a time-series numerical data, but my task is finding changes in features over time (in which those changes describe the outcome or the classes),...
21 views

How "normal" should my input data be?

When training a neural net, I understand the value in normalising the input data to have mean = 0 and stdev = 1 (standardising the data). But I often see people make the data even more "normal&...
6 views

Storing meta information from preprocessing step

Let's say I have a preprocessing step which removes some columns, replaces rare categories with the keyword "Other" and so on. All of that is done based on an initial data analysis. Now, I ...
• 103
60 views

Lost human names after 'Lemmatization' for topic modeling in python

I'm using gensim in Python for topic modeling. Currently, I have one problem. If I don't lemmatize, human names will appear as 'Most Relevant Terms for Topic,' but after lemmatization, the human names ...
56 views

What Preprocessing is Needed for Semantic Search Using Pre-trained Hugging Face Transformers?

I am building a project for my bachelor thesis and am wondering how to prepare my raw data. The goal is to program some kind of semantic search for job postings. My data set consists of stored web ...
• 23
1 vote
27 views

Getting equal distributions of data from different input sets

I am new to ML. I am trying to create a training dataset that is equally distributed between multiple lists, each of which have a different kind of data. How can I do this? I looked into ...
21 views

Data preprocessing for Multiple Linear Regression Problem

For multiple linear regression problem, I have both categorical and numerical variables in the data. I am checking the correlation for numerical variables for EDA and standardizing them by taking log. ...
13 views

Breaking out data into clusters of given size according to categorical attributes

I am trying to conduct an analysis on some product datasets. Each product has about M attributes (e.g. for M=3: (i,j,k)), whose cardinality varies wildly according to the dataset. I want to cluster ...
• 36
8 views

Classifying swarms based on 2D coordinates

The task I am trying to achieve is to classify two very differently behaving (it can be seen by the naked eye) simulated swarms. Each swarm consists of 20 individuals. I can produce data for each type ...
18 views

How can I map the amplitude of an ECG signal in relation to the time stamp?

I'm really new to programming, so can anyone suggest a way to process ECG signals from the MIT-BIH Arrhythmia database that allows me to map the amplitude and time together. Essentially, I want an ...
72 views

Reinforcement learning using univalent and multivalent heterogeneous features

Problem introduction I have a game in which human players play levels (just like the famous casual game candy crush) where each level has its own properties and its own difficulty. In said game, the ...
• 101
20 views

Predict data using Pre-Trained Classification Model

I have pre trained classification model (saved as pickle file) to predict employee attrition. My question is when I use new dataset to predict using Pickle file do I need do all preprocessing steps (...
33 views

Variable Length segments in my Dataset

I have vehicles gps dataset (time stamp, speed, acceleration, heading, latitude, longitude). This dataset is segmented to variable length of annotated batches ...
38 views

for which Machine learning models we should Discrete continuous features?

my question may be seems duplicate but its because that I couldn't find any clear and unequivocal answer for this question on the web. 1. I want to know that when should I discrete and categorize ...
• 3
178 views

Found 0 images belonging to 0 classes

Losely following this tutorial, I'm trying to apply Keras' ImageDataGenerator preprocessing on my custom object dataset. Here is the code: ...
• 103
19 views

Preprocessing in TensorFlow

Good night, I am working on a paper comparing Python libraries for machine learning and deep learning. Trying to evaluate Keras and TensorFlow separately, I'm looking for information about TensorFlow ...
1 vote
97 views

Is test data required to be transformed by training data statistics?

I am using a dataset (from literature) to build an MLP and classify real-world samples (from wetlab experiment) using this MLP. The performance of MLP on the literature dataset are well enough. I am ...
43 views

Predicting a signal based on other signals

I want to predict a signal based on other related signals, how would I go about doing this? My current approach is to do some feature extraction (in the time and frequency domain) on both the ground ...
31 views

Scaling and handling highly correlated features in tabular data for regression

I am working on a regression problem trying to predict a target variable with seven predictor variables. I have a tabular dataset of 1400 rows. Before delving into the machine learning to build a ...
• 101
45 views

BertTokenizer on custom data returns same index for all tokens

I'm trying to train Bert tokenizer on a custom dataset but when running tokenizer.tokenize on sample data, it returns the same index for every tokens which is ...
• 289
1 vote
56 views

Data preprocessing methods

Data Cleaning Data Imbalance solving (Classification) Data Smoothing (decreasing noise) Creating-deleting features from original data Data Transformation (Box-cox,Log Transform) Making Dataset ...
• 101
11 views

Interval segmentation of time series data

I have this attached time series signal (its actually from an electrostatic sensor, everytime someone walks or moves, I can see that in the signal). For the machine learning part, I would like to get ...
6 views

Ways to characterize changes in sampled data beyond variance

I'm looking for ways to characterize sampled data that captures how "monotonic" changes in the data are. For example, in the function plots below, I want a measure that differs significantly ...
• 1
1k views

Feature selection before or after scaling and splitting

Should feature scaling/standardization/normalization be done before or after feature selection, and before or after data splitting? I am confused about the order in which the various pre-processing ...
• 109
57 views

Standardization in combination with scaling

Would it be ok to standardize all the features that exhibit normal distribution (with StandardScaler) and then re-scale all the features in the range 0-1 (with <...
• 109
21 views

how to deal with features in pairwaise comparison models?

I am working on a dataset of ATP (Association of Tennis Professionals - men only) tennis games over several years. I want to predict the outcome of tennis so one way to do that is using a Bradley-...
64 views

Categorical data preprocessing for training a algorithm

I have a training dataset where values of "Output" col is dependent on three columns (which are categorical [No ordering]). ...
• 119
162 views

parallel work on KNN in python

I have a question, related to parallel work on python How I can use Processers =1,2,3... on k nearest neighbor algorithm when K=1, 2, 3,.. to find the change in time spent, speedup, and efficiency. ...
18 views

Clustering Algorithm + Euclidean Distance to find similarities

Goal: Create a tool that recommends similar players based on their statistical profile Process: (1) Standardize data (2) UMAP to reduce dimensionality (c. 50 features) (3) First-Stage Clustering: GMM ...