Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Normalization for a 2d input array

I am new to machine learning and trying to apply it to my problem. I have a training dataset with 44000 rows of features with shape 6, 25. I want to build a sequential model. I was wondering if there ...
1
vote
0answers
15 views

Data Lineage/Traceability in Pipelines

I want to collect information about: 1) from which single data signals a feature is composed in a ML pipeline and 2) what data preprocessing operations are/were executed on a data signal. Does anyone ...
1
vote
0answers
7 views

Factors in choosing a continuous encoder?

Apart from heuristically treating an encoder like a hyperparameter, how can one decide which encoder to use for continuous features? Rule out encoders that do/ don't accept negative values. Can we ...
2
votes
1answer
21 views

What is the best way to deal with image sizes in object detection using YOLO v. 4?

I am using YOLO v. 4 to perform object detection and classification on a large number of images. I do this through the Python interface. My images are all sorts of sizes and aspect ratios. Examples of ...
1
vote
1answer
21 views

Remove correlated features before or after splitting test and training set?

I want to remove highly correlated features before training my classifier. I am wondering if I should do this before or after splitting the test and training set. I don't immediately see how doing it ...
1
vote
1answer
32 views

How to impute missing value in Test Set using a custom Imputer created on training dataset

I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique. Under this technique, I replaced missing values with ...
0
votes
0answers
26 views

Impute missing values in feature column on the basis of Target column

I am working on a toy project for insurance claim prediction. In the input data for one of the feature (numeric data type) half of the values are missing. My target variable is binary which indicates ...
0
votes
2answers
45 views

Error when trying .transform for OrdinalEncoder from Scikit Learn

I'm having a lot of issues using scikit learn recently and was hoping someone could help me with my problem. I can use other methods to ordinal encode but i want to figure this one out. ...
0
votes
0answers
4 views

While Merging image datasets which of the image parameters should be prepossessed/Normalized before giving to a CNN Model?

When two datasets are merged or images of different parameters size, dimension, Format are combine which parameters of the datasets should be normalized/ pre-processed before giving it to a model?
0
votes
1answer
30 views

What will happen if almost constant values for features?

In a problem of an epidemiology dataset, is it desirable to keep the features that have almost constant values? For example, In case of the feature, type_of_residence Large for 97 percent and Small ...
0
votes
1answer
20 views

Cleaning NaNs with averages pre or post split? [duplicate]

I have a column with some NaNs in it and I want to replace those NaNs with the average/median/mode. Technically, the validation/ test data has never been seen before - so how could I include it in the ...
0
votes
1answer
31 views

Pandas replace column values by condition with averages based on a value in another column

I have a dataframe with people's CV data. Among others, there's a column with years of experience, and a column with age. Some people stated their age and experience in a way that experience > age. ...
0
votes
0answers
11 views

Bayesian Network Modeling on time series data with constant discrete features

I'm trying to model a dynamic bayesian network that can infer relationships between traffic sate of different road links over time. The main theme of mystudy is to model the relationship of change in ...
3
votes
1answer
39 views

How data are prepared during training, testing and in production?

Most of real world datasets have features with missing values. Replacing missing values with an appropriate value such as its mean, is considered as a good step in feature engineering. Some times we ...
1
vote
0answers
26 views

Is it good practice to include data cleaning or feature engineering steps in an sklearn pipeline to create a scalable pipeline?

I am working on implementing a scalable pipeline for cleaning my data and pre-processing it before modeling. I am pretty comfortable with the sklearn Pipeline ...
1
vote
1answer
22 views

What are data meta features?

I want to know what are the dataset meta features? When I google Meta Features what I get is feature selection tool called "Meta-Feature" but what I want is the definition of the dataset's ...
1
vote
1answer
17 views

Quality check for preprocessing of Text data

I have developed a pipeline for text data preprocessing with different clean up techniques like Stemming , Lemmatization, Stop words removal etc. But now the ask from the business team is to quantify ...
3
votes
2answers
118 views

Effect of Stop-Word Removal on Transformers for Text Classification

The domain here is essentially topic classification, so not necessarily a problem where stop-words have an impact on the analysis (as opposed to, say, sentiment analysis where structure can affect ...
0
votes
1answer
46 views

Different strategies for dealing with features with multiple values per sample in python machine learning models

I have a dataset which contains pregnancy, maternal, foetal and children data and I am developing a predictive machine learning model to predict adverse pregnancy outcomes. The dataset contains mostly ...
2
votes
1answer
44 views

How modelling is affected by similar feature distributions across classes?

I have grouped my dataset according to labels (good and bad customers), in an attempt to test how each feature is distributed within. Along this, I found some feature has almost exact distribution in ...
1
vote
1answer
17 views

For obejct detection, should I resize my custom images first and then start the annotation or it won't matter?

I have my custom dataset images of size (1080 x 1920) and I am trying to use yolov3 for object detection. I noticed that yolov3 model accepts an input image size of 416 x 416. So I am in confusion if ...
0
votes
0answers
19 views

With NN Model data preprocessing, is One Hot Encoding WITH PCA a good or bad idea?

With Tabular Data (containing both continuous and categorical data) preprocessing, does the One Hot Encoding of Categorical Features help or hinder the effectiveness of PCA prior to Neural Network ...
0
votes
0answers
26 views

BERT data cleaning [duplicate]

I am wondering which data cleaning steps should be performed if you want to re-fine a BERT model on custom text data. Which steps should be performed? Does it make sense to perform a stemming or ...
2
votes
1answer
22 views

Is feature scaling at all needed for a feature set with a single feature?

I understand that feature scaling is required to bring features in different magnitudes on a common scale so the model is not biased towards features with higher magnitudes. But if there is only a ...
0
votes
1answer
13 views

sine, cosine transformed cyclical features - am I losing information?

If I use sine, cosine transformation for cyclical features (e.g. weekday or hour of the day), do I lose information if the first ordinal value was 0 respectively? Assume hours of the day are encoded ...
1
vote
1answer
19 views

What's the difference between sequence preprocessing and text preprocessing in Keras?

In Keras, we mainly have three types of preprocessing, i.e., sequence preprocessing, text preprocessing, and image preprocessing. However, for me, I think the meanings of the word "sequence" ...
0
votes
2answers
31 views

What is the best way to treat datetime in the preprocessing step of machine learning

I have two datetime columns in my dataset. What I have done so far I have extracted year, ...
0
votes
1answer
17 views

Hand preprocessing data

I am working with a relatively small (~1000 samples) dataset that has some very messy text data (i.e lots of things missing, no real structure, etc.). I am trying to preprocess it and just going ...
1
vote
1answer
14 views

How to conclude the generality of any classification methods?

Suppose a classification task A, and there exist a lot of methods $M_1, M_2, M_3$. The task $A$ is measured by a consistent measure. For instance, the task A can be a binary classification. In this ...
2
votes
2answers
61 views

Repeated features in Neural Networks with tabular data

When using algorithms like linear regression or least-squares methods, having repeated or highly correlated features can be harmful for the model. For tree based models, they are generally not too ...
0
votes
1answer
19 views

Log scaling whole dataset

I am log scaling my whole dataset in log base 10. When I do this I get -infitity for the minimum value. I am wondering how I can get rid of this -infinity? I have been advised to add a small value ...
1
vote
0answers
30 views

How to control for Co-variate shift in test data set compared to train data for regression task?

I am working on a regression project. But I am facing the problem of covariate shift in features due to time delay.Test data was collected a year later due to which there has been some change in ...
1
vote
0answers
17 views

Python : data type handling by sklearn and impact on memory usage and performance

I am currently working on reducing the memory usage of my data (Pandas DataFrame). This is going quite well : downcasting floats into smaller floats, integers into smaller ones and transforming string ...
0
votes
2answers
50 views

Rolling window features for multiclass classification [closed]

I'm doing a multiclass classification and data is considered as not being a time-series. Working on a feature engineering and trying to solve the problem with classic KNN, RF, boosting etc. I'm ...
1
vote
0answers
30 views

What can help decrease outliers' influence on non-tree models?

I have a feature with all the values between 0 and 1 except few outliers larger than 1. I am trying to collect all the methods that can help to decrease outliers' influence on non-tree models: ...
3
votes
1answer
36 views

Smart sentence segmentation not splitting on abbreviations

Sentencer from SpaCy and NLTK does not catch the fact that typical abbreviations (e.g. Mio. for Million in German) and the ...
1
vote
1answer
47 views

Use heading in Neural Network model

I am working on a prediction model where I must find out the destination of a boat based on its actual coordinates and heading (compass direction) : ...
1
vote
1answer
26 views

difference between scaling/normalizing data at a specific step

I am using the MinMaxScaler normalization method, however I have seen various ways that this can be done, I want to know if there is any actual difference between the following: 1. Standardizing/...
1
vote
0answers
10 views
0
votes
0answers
18 views

Reducing training data to a smaller subset for neural network regression task

I have a large dataset of several billion inputs of categorical attributes. The neural network performs a regression task. Since I want to train the model, but not always use all of the data, how can ...
1
vote
0answers
10 views

Preparing training data for crop species classification from drone images

I have high resolution multispectral (R,G,B,NIR,RE bands) images of a field taken from MicaSense RedEdge mounted on a drone. There are various species of crops planted. I want to classify the crops or ...
1
vote
2answers
76 views

is it better to correlate and encode or encode and correlate?

I have one doubt like is it better to perform label encoding and check for the correlation or should I 1st perform correlation and do label encoding? Because when I tried it both ways I'm getting ...
1
vote
1answer
19 views

How to handle fixed values for variables in pre-processing

I have a dataset which contains few variables whose values do not change. Some of the variables are non-numeric (for example all values for that variable contain the value 5) and few variables are ...
1
vote
0answers
14 views

Redo preprocess on unknown row

I'm trying to write a script to get the most similar rows in a certain dataframe, based on a single row. Using scikit-learn The method I need is ...
4
votes
2answers
76 views

If there are no missing values in our training set, should we accommodate missing values in an unseen test set?

My training data has no missing values. I'm unsure whether or not I should fit say, imputation, on the training set so that I can accommodate possible missing values on the test set, because the test ...
0
votes
0answers
18 views

Should the column be kept from which range column was extracted?

I'm creating a Neural Network model to predict the price range of laptop. Input would be the configurations of that laptop. But the price range column was created from the price column. If I keep the ...
0
votes
1answer
18 views

How to expand abbrevations in text during preprocessing?

Im doing preprocessing on english text data. I have some domain specific abbreviations, for which i'm maintaining internal dictionary with key-value pairs. The problem i'm facing is the text has ...
1
vote
3answers
101 views

How to process the hyphenated english words for any nlp problem?

Im doing preprocessing on english text dataset. I encounter hyphenated words like 'well-known'. Will it be useful if I remove the hyphen as special character and treat it as a single word 'wellknown' ...
4
votes
3answers
269 views

Reducing the size of a dataset

I am trying to classify gestures. I am using Python's scikit learn library classification algorithms for that. I have collected depth images for this purpose. 200 samples are collected for each ...
0
votes
0answers
18 views

What is the suggested way to create features (Mel-Spectograms) from speech signal for classification with ResNet?

At the moment I have this piece of code which cuts a Spectogram into fixed length tensors: ...

1
2 3 4 5
8