Questions tagged [preprocessing]

Data preprocessing is a data mining technique that involves transforming raw data into a better understandable or more useful format.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

Bibliography on the preprocessing of data (in a statistical perspective) in data science

In data science, very often we're given a data set which contains duplicate rows, missing values, outliers, etc. Most of the information on how to deal with these situations (from a statistical ...
0
votes
0answers
5 views

PrepFlow - Data Column renaming for same named columns

If I have 5 joins in the my prep flow, and I'm finding that all 5 have some common fields like how does one determine which _id belongs to which join? do I need to manually rename each one with a more ...
2
votes
1answer
18 views

Data preprocessing framework/library alternatives

I am currently working on some python machine learning projects that are soon to be deployed to production. As such, in our team we are interested in doing this the most "correct" way, ...
1
vote
0answers
31 views

Can numerical encoding really replace one-hot encoding?

I am reading these articles (see below), which advocate the use of numerical encoding rather than one hot encoding for better interpretability of feature importance output from ensemble models. This ...
1
vote
0answers
16 views

How to convert varians of a character to the same character in python? [closed]

How to convert variants of a character to a same character in python. For example, the character a with the U+0061 Unicode has ...
0
votes
2answers
19 views

How does Keras Tokenizer choose tokens given a sentence?

I tried to find the answer to this question but I can't find anything, so I ask here: How does Keras Tokenizer choose tokens given a sentence of words ? To be more precise with what I want to know, ...
0
votes
0answers
12 views

Training a ML model with a table for each observation

I have several csv's which are inputs for a row of outputs. A sample input dataset can look something like this: whose output would be as follows: The task is to train a model by reading each csv as ...
0
votes
1answer
34 views

Two-parametric transformation of Box-Cox vs Yeo–Johnson transformation

I choose which transformation to use for my data (data contains both positive and negative values). Wikipedia says the following: The Yeo – Johnson transformation allows also for zero and negative ...
0
votes
0answers
10 views

Are there differences in preprocessing nominal vs ordinal vs interval vs ratio data

I wonder are there significant differences that ought to be known when preprocessing nominal vs ordinal vs interval vs ratio. Intuitively, it seems like encoding ordinal values should be performed ...
0
votes
2answers
42 views

How to remove rows from a data frame that have special character (any character except alphabet and numbers)

How to remove rows from a data frame that have special character (any character except alphabet and numbers) I have some unwanted labels which seems to be useless but I want to remove them, this are ...
1
vote
0answers
14 views

Is it possible to strip/supress embedded bounding boxes from images?

I'm doing a project that needs to use this company's CCTV image samples to build my own object detection model, although the provided images all have red/green bounding boxes baked into the jpg's with ...
1
vote
0answers
21 views

How to predict data from sequence of sequences of variable size?

input data ...
0
votes
0answers
9 views

XGBoost regressor with weird output value

A regression task that the y value’s range is around a * 10^-5 to b * 10^-4 and trying to use XGBoost to handle this question. And the weird thing is, when finished model training, the model ...
2
votes
1answer
33 views

Smoothing target variable

I am training a regression model (using quantile regression forest) to forecast crop yield deviations from trend (residuals) using weather variables with different lag times. Trying to improve the ...
2
votes
1answer
32 views

Some questions about supervised learning, model evaluation and preprocessing [closed]

I've been trying to employ some basic techniques of supervised learning on a dataset that I have and I have several questions about the overall procedure (i.e. data preprocessing, model evaluation etc)...
0
votes
1answer
20 views

Applying standardization using ImageDataGenerator

I have a multiclass image dataset ( 8 classes) that is divided as follows, the main folder is called training and I have 8 subfolders with each subfolder for one class. I know how to perform data ...
0
votes
1answer
30 views

Custom preprocessing using piplines

I have searched a lot for this issue but unfortunately came up with nothing. Usually in a ML model, during preprocessing, we use Pipelines and ...
1
vote
1answer
19 views

Preprocessing for the final model to be deployed

Typically for a ML workflow, we import the data (X and y), split the X and ...
0
votes
0answers
25 views

NLP conversation data - Pre-processing steps

I have text data, so data that has been transcribed from conversations from employees to customers. So each call has a recording that has been transcribed to text. I am looking to do some analytics ...
0
votes
0answers
9 views

How to calculate relation function of State and Action in delayed action effect Environment?

I'm trying to calculate corr-coef(or other good relation function) of State and Action in 'delayed action effect' Envrironment. In this environment, agent observes states, then it returns action and ...
2
votes
1answer
25 views

Best way to preprocess data

I need to create a machine learning model to predict if a structure is an hotel or an apartment. I have a dataset structured as well: ...
1
vote
0answers
32 views

Identify MCAR, MNAR and MAR in the data

If I have missing values in a dataset, I can't just blindly impute them with mean/median/mode or any other technique. I have to identify what kind of missing values they are, namely: MCAR (missing ...
0
votes
0answers
14 views

Checking Skweness for each column in Dataset

Is checking Skweness for each column before feeding it to the algorithm a mandatory step while preprocessing our dataset or on what conditions does checking skewness depends? Currently I am working on ...
1
vote
1answer
41 views

test data is not a good representation of train data

I have predefined train and test sets. On generating some statistics like value_counts and checking the unique values, I feel that there is a 'lot' of difference between the distributions of the ...
0
votes
0answers
34 views

How can I balance sentence data for NLP tasks

I have been given a task to train the SVM model on conll2003 dataset for Named Entity "Identification" (That is I have to tag all tokens in "Statue of Liberty" as named entities ...
0
votes
1answer
26 views

Query regarding the 'Data type' of features in Machine Learning

Should all the features in a dataset be converted to the same data type? For instance, if all the features have numerical values, some int & some float, should they all be converted to float? What ...
0
votes
1answer
35 views

Working with three types of data: numeric (integer, floats), images, and text for prediction

So I have three types of data (in title) and am wondering how I can combine the data. The target is numeric (price). My idea is to perform feature extraction on both the images and text, which would ...
1
vote
0answers
42 views

When use standardization, normalization or both?

I have a dataset with variables with different scales as shown in the figure below. I need to group individuals together and I'm testing algorithms like Kmeans and DBScan. In all tests I'm extracting ...
1
vote
0answers
18 views

How should you preprocess the data before K-fold validation?

I often see Kaggle notebook authors preprocessing the entire training data prior to splitting it for K-fold validation, but does this have a risk of leaking information into the validation set each ...
1
vote
2answers
86 views

Do we need to pre-process both the test and train data set?

I've been given 2 datasets , and there are missing values in both the test and training data set. Do I need to pre-process test.csv also or is it only for train.csv?
0
votes
0answers
14 views

When to do tokenization and does my output need tokenization after stemming?

I am working on sentiment analysis project , where there are various customer reviews. So I am trying to clean those reviews. So first thing i did is removing special characters, white spaces, ...
0
votes
0answers
26 views

What exactly "subset" is in "tf.keras.preprocessing.text_dataset_from_directory"?

so I'm following the official keras tutorial here. However I couldn't really understand the subset and validation_split ...
0
votes
1answer
22 views

How to deal with data having 0 values in many columns?

I am trying to implement logistic regression but the dataset that I have have many columns with skewed data and most of them have 0 as values. I also the skewness of data for many columns its going ...
1
vote
2answers
61 views

data preprocessing: missing labels

I have a classification task to predict {0,1} labels, but in my dataset I only have data with 1 labels. Class indicates that the buyer has made a purchase of some product. Features: user id, product ...
3
votes
3answers
44 views

Organising the preprocessing of a dataset that will be consumed by multiple models

In a data science project, data is typically preprocessed. We also build, test, and select different models. Models also come with their own preprocessing requirements that can vary greatly from model ...
0
votes
0answers
26 views

Does this make data leakage in time series? # need help for understanding time series data

Does this make data leakage in time series? I already read this, data leakage when scaling time series Data leakage is when information from outside the training dataset is used to create the model. ...
1
vote
1answer
37 views

scikit-learn OneHot returns tuples and not a vectors

First I do a label encoding to all the columns that are strings so they will be numeric. After that, I take just the columns with the labels, convert them to np array, reshape, and convert them to one-...
0
votes
1answer
19 views

ML for data processing. What are the options?

Currently I am working on improving a stage on a data processing pipeline. The source data has a large number of fields and is getting normalized into a simpler entity. This entails that in many cases ...
0
votes
0answers
22 views

StandardScaler on data which increase over time

I am trying to apply standard scaler on my data for classification prediction. But one of the feature will increase over time e.g. lifetime count, days since join Should I apply the standard scaler ...
1
vote
2answers
64 views

Dealing with high number of NAs in a classification problem

I am working on a classification problem. The dataset dimension is as 187,643 x 203. The first column contains class labels with no NA. The rest of dataset are frequency data and could be anything ...
0
votes
1answer
148 views

Splitting before tfidf or after?

When should I perform preprocessing and matrix creation of text data in NLP, before or after train_test_split? Below is my sample code where I have done ...
0
votes
0answers
12 views

Naive Bayes unable to detect preprocessing techniques from data

I'm testing out different preprocessing techniques on mulit-class classification problems. I've used multiple algorithms, but the only algorithm that giving me trouble is Naive Bayes. ...
0
votes
1answer
63 views

Preprocessing , EDA , and Feature Engineering

What is the difference between EDA, Feature Engineering, and Preprocessing? The main purpose is to make the raw data suitable for modeling. In EDA, we are cleaning the data and so does the ...
0
votes
1answer
18 views

Encoding concept for categorical data - pick one for all the columns or different for different kinds in the same df

[Beginner here] If dataset contains - both ordinal, nonordinal (few categories) & nonordinal (multiple categories > 30). Is one supposed to pick one to encapsulate of all such situations or ...
0
votes
0answers
14 views

Outliers capping is leading to generation of new duplicates during data pre-processing

So as the title suggest, I removed duplicates which were around 5% to the data but after outliers capping, new duplicates got generated in huge amount (~8%) so what should I do in this case? I'm going ...
1
vote
1answer
21 views

If we replace all missing values with "unknown" or "-∞", what problem will we encounter?

I am reading Han,Kamber,Pei's data mining book and I stumbled upon a section called "data cleaning". It tells we can use a global constant like "unknown" or "-∞" to ...
0
votes
0answers
34 views

What do we mean by permissible transformations in types of attributes-:nominal,ordinal,interval,ratio?

I am studying data mining and I stumbled upon types of attributes. They are Nominal Ordinal Interval Ratio Data mining book by Tan,Steinbech,Kumar says Permissible transformations for-: nominal-:...
1
vote
1answer
59 views

How to increase the accuracy of an imbalanced dataset (not precision)?

There's an imbalanced dataset in a Kaggle competition I'm trying. The target variable of the dataset is binary and it is biased towards 0. 0 - 70% 1 - 30% I tried several machine learning algorithms ...
1
vote
3answers
78 views

How does missing data occur

I am new to ML. I found that one of the preprocessing steps is to handle missing data. My query is Is there a way to understand nature of missing data I can see that the mostly missing data is ...
0
votes
1answer
189 views

Separate numerical and categorical variables

I have a dataset (42000, 10) which contains 7 categorical features and 3 numerical. I would like to separate both the numerical and categorical features into 2 different data frames i.e I would like 2 ...

1
2 3 4 5
9