Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
0 votes
0 answers
42 views

Reinforcement learning using univalent and multivalent heterogeneous features

Problem introduction I have a game in which human players play levels (just like the famous casual game candy crush) where each level has its own properties and its own difficulty. In said game, the ...
user avatar
  • 1
0 votes
1 answer
16 views

Predict data using Pre-Trained Classification Model

I have pre trained classification model (saved as pickle file) to predict employee attrition. My question is when I use new dataset to predict using Pickle file do I need do all preprocessing steps (...
user avatar
0 votes
0 answers
30 views

Variable Length segments in my Dataset

I have vehicles gps dataset (time stamp, speed, acceleration, heading, latitude, longitude). This dataset is segmented to variable length of annotated batches ...
user avatar
0 votes
0 answers
13 views

for which Machine learning models we should Discrete continuous features?

my question may be seems duplicate but its because that I couldn't find any clear and unequivocal answer for this question on the web. 1. I want to know that when should I discrete and categorize ...
user avatar
  • 3
0 votes
1 answer
20 views

Found 0 images belonging to 0 classes

Losely following this tutorial, I'm trying to apply Keras' ImageDataGenerator preprocessing on my custom object dataset. Here is the code: ...
user avatar
  • 103
0 votes
1 answer
18 views

Preprocessing in TensorFlow

Good night, I am working on a paper comparing Python libraries for machine learning and deep learning. Trying to evaluate Keras and TensorFlow separately, I'm looking for information about TensorFlow ...
user avatar
  • 3
1 vote
1 answer
37 views

Is test data required to be transformed by training data statistics?

I am using a dataset (from literature) to build an MLP and classify real-world samples (from wetlab experiment) using this MLP. The performance of MLP on the literature dataset are well enough. I am ...
user avatar
0 votes
0 answers
17 views

Predicting a signal based on other signals

I want to predict a signal based on other related signals, how would I go about doing this? My current approach is to do some feature extraction (in the time and frequency domain) on both the ground ...
user avatar
0 votes
0 answers
9 views

Scaling and handling highly correlated features in tabular data for regression

I am working on a regression problem trying to predict a target variable with seven predictor variables. I have a tabular dataset of 1400 rows. Before delving into the machine learning to build a ...
user avatar
  • 101
0 votes
0 answers
12 views

BertTokenizer on custom data returns same index for all tokens

I'm trying to train Bert tokenizer on a custom dataset but when running tokenizer.tokenize on sample data, it returns the same index for every tokens which is ...
user avatar
  • 259
1 vote
1 answer
52 views

Data preprocessing methods

Data Cleaning Data Imbalance solving (Classification) Data Smoothing (decreasing noise) Creating-deleting features from original data Data Transformation (Box-cox,Log Transform) Making Dataset ...
user avatar
  • 45
0 votes
0 answers
10 views

Interval segmentation of time series data

I have this attached time series signal (its actually from an electrostatic sensor, everytime someone walks or moves, I can see that in the signal). For the machine learning part, I would like to get ...
user avatar
0 votes
0 answers
6 views

Ways to characterize changes in sampled data beyond variance

I'm looking for ways to characterize sampled data that captures how "monotonic" changes in the data are. For example, in the function plots below, I want a measure that differs significantly ...
user avatar
  • 1
0 votes
1 answer
137 views

Feature selection before or after scaling and splitting

Should feature scaling/standardization/normalization be done before or after feature selection, and before or after data splitting? I am confused about the order in which the various pre-processing ...
user avatar
  • 109
0 votes
2 answers
41 views

Standardization in combination with scaling

Would it be ok to standardize all the features that exhibit normal distribution (with StandardScaler) and then re-scale all the features in the range 0-1 (with <...
user avatar
  • 109
0 votes
0 answers
19 views

how to deal with features in pairwaise comparison models?

I am working on a dataset of ATP (Association of Tennis Professionals - men only) tennis games over several years. I want to predict the outcome of tennis so one way to do that is using a Bradley-...
user avatar
0 votes
1 answer
24 views

Categorical data preprocessing for training a algorithm

I have a training dataset where values of "Output" col is dependent on three columns (which are categorical [No ordering]). ...
user avatar
  • 19
0 votes
1 answer
39 views

parallel work on KNN in python

I have a question, related to parallel work on python How I can use Processers =1,2,3... on k nearest neighbor algorithm when K=1, 2, 3,.. to find the change in time spent, speedup, and efficiency. ...
user avatar
0 votes
0 answers
16 views

Clustering Algorithm + Euclidean Distance to find similarities

Goal: Create a tool that recommends similar players based on their statistical profile Process: (1) Standardize data (2) UMAP to reduce dimensionality (c. 50 features) (3) First-Stage Clustering: GMM ...
user avatar
0 votes
1 answer
26 views

How to provide Intentional Bias towards recent examples in Text Classification?

I have trained an XGBClassifier to classify text issues to a rightful assignee (simple 50-way classification). The source from where I am fetching the data also provides a datetime object which gives ...
user avatar
0 votes
0 answers
54 views

AttributeError: 'MissingValues' object has no attribute 'to_list' while i am using LabelEncoder() from sklearn

the same process when done in python 3.9 with pandas and csv dataset it is working fine but how should i use label encoder on geopandas dataframe with python 3.6 and sklearn version 0.24.2.
user avatar
0 votes
0 answers
9 views

Should I inpute the missing values before the train-validation split?

validation is suppose to provide an unbiased evaluation of a model fit on the training data. In that case inputation before the training-validation split could cause an indirect data leakage because ...
user avatar
1 vote
0 answers
14 views

Interpretation of the results of the Elbow and K-means

I have the following dataset (after scaling) which contains 5 features: : My objective is to cluster this data using an unsupervised ML model. After using the Elbow method, I get 2 clusters as below: ...
user avatar
0 votes
0 answers
14 views

Which variables to make stationary, dependent or target?

I'm trying to understand which part of my data should I transform such that it's stationary, from the assumption that ML models perform best when data is stationary. Does it make sense to transform ...
user avatar
0 votes
0 answers
10 views

Single to Multiple data feature

How do you deal with multiple to single data feature? I have an entry that is connected to a few stacked layers. There are 2 types of layers and each entry has a different number and arrangement of ...
user avatar
  • 1
0 votes
0 answers
14 views

Clarify normalization methods

I'm following this guide on detecting anomalies using autoencoders. The section titled "Normalising & Standardising" seems to be describing normalization in terms of scaling and shifting ...
user avatar
  • 111
0 votes
0 answers
6 views

Best way to get independent rows in your data set?

I have contract data which has some repeats of accounts so some of the rows are dependent on one another. How can I deal with this? I was thinking to put each account's data into a single row, but ...
user avatar
  • 1
0 votes
1 answer
38 views

my all categories convert to NaN or 'missing' ; what's the problem?

Hello, I have a dataFrame and one of features is categorical and I want to convert that to ordinal category(including a category for missing values) but in the last cell as you see it assumes all of ...
user avatar
  • 3
0 votes
0 answers
7 views

Classification using STFT for multiple categories of signal samples

I have a collection of signals (IQ wav) split up into ~2s samples of sampling rate 2MHz, and can collect the STFT information from these samples through the following code: ...
user avatar
  • 111
0 votes
0 answers
8 views

What exactly is the "currency" dimension in data profiling?

...
user avatar
  • 133
0 votes
0 answers
93 views

Pass information between pipeline steps in sklearn

I am working on a simple text generation problem with LSTMs. To make the preprocessing more compact and reproducible, I decided to implement everything in sklearn fashion, using custom sklearn ...
user avatar
  • 259
0 votes
0 answers
32 views

Using differencing multiple times for making dataset stationary

When I do differencing to my dataset I am having a lot of zeroes and it causes my prediction to go wrong. But when I use it multiple times, my dataset is having minus values but still, at least my ...
user avatar
0 votes
2 answers
35 views

Why label encoding before split is data leakage?

I want to ask why Label Encoding before train test split is considered data leakage? From my point of view, it is not. Because, for example, you encode "good" to 2, "neutral" to 1 ...
user avatar
  • 73
0 votes
1 answer
81 views

Dynamic creation of sklearn pipeline

I am trying to create an automatic pipeline builder functionality that takes into account a large set of conditions such as the existence of missing values, the scale of numerical features, etc., and ...
user avatar
  • 259
0 votes
0 answers
12 views

Time Series Data Missing Value Treatment

I have an hourly time series data for a solar plant which covers 3 years (2019, 2020, 2021). I have a categorical feature named WWCode which has 54 unique values. WWCode is actually a weather ...
user avatar
0 votes
1 answer
12 views

Replacing missing missing view count when photo

I'm currently analyzing a dataset of posts on Facebook. Some are videos and others are photos. One of the features is view_count which has missing values every time the post is a photo. How can I ...
user avatar
  • 93
-1 votes
2 answers
241 views

Vectorized String Distance

I am looking for a way to calculate the string distance between two Pandas dataframe columns in a vectorized way. I tried distance and textdistance libraries but they require to use df.apply which is ...
user avatar
  • 121
0 votes
1 answer
229 views

__init__() takes 1 positional argument but 4 were given sklearn standard scaler error

I defined a class like below: ...
user avatar
0 votes
0 answers
16 views

How to identify the limiting factor in my text classification model?

I am working on building a comments classification model, with about 2500+ comments (varying in length from 5 to ~110 words) and 11 categories (yes I understand the ratio is quite bad). So far, I have ...
user avatar
0 votes
1 answer
16 views

Is It Okay To Do 0-1 Scaling Then Divide By The Standard Deviation?

If am understanding stuff correctly, if I have a df I can first do 0-1 scaling on it to get equal ranges while preserving the data series's original means and standard deviations and then once I ...
user avatar
1 vote
1 answer
32 views

PowerTransformer Producing Unexpected Result for Just One Column

I'm doing some preprocessing on my training data before fitting it to a model. Upon checking the results, there is one column that is returning 0 rather than 1 for the standard deviation. (all columns ...
user avatar
  • 11
0 votes
1 answer
29 views

Where can i learn time series data forcasting/analysis?

I would like to learn time series data analysis and forecasting. I am knowledgeable in machine learning and have a good knowledge of deep learning (including RNN's, LSTM). I came to know that time ...
user avatar
1 vote
1 answer
49 views

Why is an ML algorithm performing better with correlated features, than the one with uncorrelated ones?

I have a dataset with all numerical values. Since the features were not many, I created more by multiplying pairs of each other. This created some highly correlated features, as expected. Now, I ...
user avatar
0 votes
2 answers
184 views

How to remove outliers properly?

I was wondering what is the best practice for removing outliers from data. Plotting a boxplot for each feature (column of the dataset) and removing data that fall outside the whiskers seems like a ...
user avatar
  • 73
5 votes
4 answers
3k views

In ML why selecting the best variables?

Almost all ML notebooks out there have a section where they select the best features to use in the model. Why is this step always there ? How bad can it be to keep a variable that is not correlated ...
user avatar
  • 121
0 votes
0 answers
15 views

Bibliography on the preprocessing of data (in a statistical perspective) in data science

In data science, very often we're given a data set which contains duplicate rows, missing values, outliers, etc. Most of the information on how to deal with these situations (from a statistical ...
user avatar
0 votes
0 answers
8 views

PrepFlow - Data Column renaming for same named columns

If I have 5 joins in the my prep flow, and I'm finding that all 5 have some common fields like how does one determine which _id belongs to which join? do I need to manually rename each one with a more ...
user avatar
  • 101
2 votes
1 answer
38 views

Data preprocessing framework/library alternatives

I am currently working on some python machine learning projects that are soon to be deployed to production. As such, in our team we are interested in doing this the most "correct" way, ...
user avatar
1 vote
0 answers
33 views

Can numerical encoding really replace one-hot encoding?

I am reading these articles (see below), which advocate the use of numerical encoding rather than one hot encoding for better interpretability of feature importance output from ensemble models. This ...
user avatar
1 vote
0 answers
17 views

How to convert varians of a character to the same character in python? [closed]

How to convert variants of a character to a same character in python. For example, the character a with the U+0061 Unicode has ...
user avatar

1
2 3 4 5
10