Questions tagged [data-imputation]

Data imputation is the process of replacing missing data with substituted values. This could involve statistically representative data filling (e.g. local averages) or simply replacing the missing data with encoded values (e.g. replace NaNs with zeros).

Filter by
Sorted by
Tagged with
2
votes
1answer
52 views

Can data leakage be sometimes acceptable?

I have recently started using kaggle and I have stumbled on a few examples of practices I would consider do be data leakage. Many of them were done by people well established on the platform and I ...
4
votes
5answers
136 views

Dropping columns or inputing numbers

After looking at the various different ways of inputting data to replace NaN in a dataset vs. dropping observations or columns based on a threshold, the right ...
1
vote
2answers
70 views

Check if a particular value is a datetime and assigning a particular column value in pandas

I have a pandas data frame that contains a partially corrupted data field as below. It has numbers (which are not a date) or nans. The real data frame has an incredibly large number of rows as well. I ...
0
votes
0answers
11 views

Linear models: Imputing missing not at random

This question is a continuation of a similar question for linear models instead of Tree-based model. Given that linear models (e.g. lasso, ridge, Linear regression, elastic net, etc.) can't handle ...
1
vote
1answer
97 views

XGBoost - Imputing Vs keeping NaN

What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at ...
0
votes
0answers
25 views

How to fill missing latitude and longitude values in time series data?

I have time-series data like this: date longitude latitude 01/01/2010 -5.42766 107.5784 02/01/2010 -6.42728 104.5245 07/01/2010 -7.42702 105.5816 14/01/2010 -4.42728 99.57834 17/01/2010 -6.41523 ...
1
vote
2answers
113 views

Using scikit-learn iterative imputer with extra tree regressor eats a lot of RAM

I'm imputing a table around 150K by 60 floats and has about 45% missing values, I'm using ExtraTreeRegressor with IterativeImputer ...
0
votes
1answer
17 views

Impute missing value: transpose or not?

I'm building a model that fills the missing values from a Dataframe that contains the number of visitors for different stores, each day: day store_a store_b store_c 2021-01-01 100 200 300 2021-01-...
0
votes
0answers
10 views

How to impute when you have multiple variables not important to the column with the missing values?

I have a dataset with 18000 rows and 192 columns. I have a specific column with more than 2000 rows missing. I've tried some types of imputations but they just take to long or just seem not good ...
2
votes
1answer
61 views

kNN for non-ordinal variables

kNN is a distance-based method, so it requires the input to be in numerical form. I was wondering if it is possible to use kNN imputer for non-ordinal categorical variables (like color). Since the ...
1
vote
1answer
28 views

Is there potentially data leakage during imputation for time-varying sensor data?

I have a time-varying dataset that contains some missing data. I have sensors that continuously monitor some properties at evenly-spaced intervals and I would like to impute the missing values using ...
0
votes
0answers
32 views

How can I compare imputation techniques on a dataset with sci-kit learn?

I have a dataset data that has missing values. I am trying two ways of imputing these values, but I would like to compare them. In the first method I am using a ...
1
vote
1answer
69 views

How to impute missing value in Test Set using a custom Imputer created on training dataset

I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique. Under this technique, I replaced missing values with ...
0
votes
0answers
91 views

Impute missing values in feature column on the basis of Target column

I am working on a toy project for insurance claim prediction. In the input data for one of the feature (numeric data type) half of the values are missing. My target variable is binary which indicates ...
0
votes
1answer
135 views

K-Fold cross validation and data leakage

I want to do K-Fold cross validation and also I want to do normalization or feature scaling for each fold. So let's say we have k folds. At each step we take one fold as validation set and the ...
1
vote
1answer
14 views

Dropping Missing Observations under MAR Assumption

Some of the outcome data in my data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing and ...
1
vote
0answers
30 views

How to handle large systematic missing data in time series?

I have this time series, where on the weekends, the dependent variable values are missing. It's only a time series, I do not have any exogenous regressors/features. The dependent variable value is an ...
0
votes
1answer
72 views

sklearn KNN imputation

Can I use sklearn's KNN imputer to fit the model to my training set and impute missing values in the test set using the neighbours from training set ? Is it allowed ? Or , Should I only fit and ...
3
votes
2answers
54 views

Missing value Imputation in dataset

I have two separate files for Testing and Training. In the training data, I am dropping rows that contain too many missing values . But , In the test data , I cannot afford to drop the rows so I have ...
1
vote
2answers
461 views

sklearn - How to create a sequential pipeline

Update: The examples in this post were updated I am reposting this question here after not getting a clear answer in a previous SO post I am looking for a help building a data preprocessing pipleline ...
0
votes
0answers
111 views

sklearn SimpleImputer using mean from different column groups?

I'm looking at the SimpleImputer, in particular in here, and I would like to do the imputation on different columns. My data has 3 different sample groups, and I would like to do ...
1
vote
1answer
332 views

Using sklearn knn imputation on a large dataset

I have a large dataset ~ 1 million rows by 400 features and I want to impute the missing values using sklearn KNNImputer. Trying this off the bat I hit memory problems, but I think I can solve this by ...
0
votes
0answers
23 views

If a single element is missing more than 50% of its feature values, should you just remove it?

I have a simple supervised machine learning problem. My training matrix is MxN, where M is the number of records and N is the number of features. I have 600,000 complete patient records and 300,000 ...
1
vote
1answer
40 views

In supervised learning, are more data entries always better?

I am doing a supervised learning problem and have 600,000 rows of data. I divided it into a training and test set and achieved a high accuracy that I was happy with. However, I had thrown away 300,000 ...
0
votes
1answer
44 views

How to handle sparsely coded features in a dataframe

I have a dataset that contains information regarding diabetes patients, like so: ...
1
vote
2answers
283 views

KNN Imputation utilize mean or mode?

In my current project, I am doing KNN imputation with K = 5 and I am using sklearn.impute.KNNImputer. I have a mix of continuous and nominal variables(encoded as 0/1 or ordinal ones that have been ...
1
vote
1answer
44 views

How can we use mean imputation without violating feature correlation?

Mean imputation is generally bad practice because it doesn’t take into account feature correlation. Imagine we have a table showing age and fitness score and imagine that an eighty-year-old has a ...
1
vote
1answer
59 views

Dealing with missing data

I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both ...
1
vote
1answer
722 views

How to impute using simple imputer (custom function)

I am imputing my data using simple imputer from sklearn. i want to test many different ways of applying transformations to the data. i.e for logisitcic regression i would like to remove nans and ...
2
votes
4answers
160 views

Handling categorical missing values ML

I have gone through this regarding handling missing values in categorical data. Dataset has about 6 categorical columns with ...
1
vote
2answers
458 views

What is the point of using MissingIndicator in Scikit-learn?

I have recently discovered Sklearn's MissingIndicator but still wondering how could it improve the usual machine learning work. Clear that ...
1
vote
0answers
17 views

Fit model function out defined data range

I have asked this on SO but it has not been well accepted because it seems to be more about data science than programming. Let's say I have a set of data ...
1
vote
2answers
107 views

Right order for Data preparation in Machine Learning

For the below mentioned steps of data preparation Outlier detection/treatment Data imputation Data scaling/standardisation Class balancing There are two sub questions Should each of these steps ...
10
votes
5answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
0
votes
1answer
110 views

How to impute right-censored data

I have a dataset of vectors representing movement with various characteristics. Some vectors represents the movement that was stopped by external factor and therefore, observed value for length of ...
-1
votes
1answer
311 views

How to handle time series missing values

I have a database of thermal consumption of 100 buildings. Each file has two columns, one is timestamp and the other is usage. My task is to build a prediction model for forecasting the usage for the ...
2
votes
2answers
621 views

What do I do when my column has 50% data missing?

I have a data set i am pre-processing. However in my categorical columns (3 of them) i have "??" in it's place. They constitute 50% of the data. In fact 3 columns have this. My question is how should ...
1
vote
1answer
21 views

Dealing with issues in “test” predictons for single “items” (null values, standardization in place, etc)

I know this is kind of a broad question but I have tried to scour both this forum and the internet in general to no avail for this particular situation. So imagine I have a model trained for which, ...
0
votes
0answers
61 views

What is the difference between “fit_transform” and “transform” methods when using “SimpleImputer”? [duplicate]

I have following code, I am not able to understand the difference between use of fit_transform() and transform() method in this ...
7
votes
3answers
1k views

Retrieve dropped column names from `sklearn.impute.SimpleImputer`

The SimpleImputer class takes pandas dataframes and returns unlabeled numpy arrays. Which means that the SimpleImputer drops ...
2
votes
1answer
74 views

How to implement single Imputation from conditional distribution?

In [*] page 264, a method of drawing a missing value from a conditional distribution $P(\bf{x}_{mis}|\bf{x}_{obs};\theta)$ which is defined as: I did not find any code implementation of this ...
1
vote
0answers
138 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
0
votes
2answers
7k views

How can I replace outliers with maximum non-outlier value?

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no ...
6
votes
2answers
498 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
9
votes
2answers
4k views

Which comes first? Multiple Imputation, Splitting into train/test, or Standardization/Normalization

I am working on a multi-class classification problem, with ~65 features and ~150K instances. 30% of features are categorical and the rest are numerical (continuous). I understand that standardization ...
3
votes
1answer
1k views

Advice on imputing temperature data with StatsModels MICE

This may be a dumb question but I can't figure out how to actually get the values imputed using StatsModels MICE back into my data. I have a dataframe (dfLocal) with hourly temperature records for ...
1
vote
2answers
1k views

Is it right to impute Train and Test set?

I am experimenting with a dataset and I have a couple of columns with high cardinality. So, I performed mean target encoding (given that my dataset had more than 50000 observations). But, before doing ...
4
votes
3answers
1k views

What predictive model to use to impute Gender?

My data looks like this: birth_date has 634,990 missing values gender has 328,849 missing values Both of these are a ...
2
votes
3answers
111 views

Machine Learning, Imputing values that should be blank

Sometimes data sets contain variables that indicate the presence of an event and the value that represented the event. As an example say a teacher wants to predict the grades of his students. Some of ...
3
votes
1answer
1k views

Target Encoding: missing value imputation before or after encoding

I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values. Let's say I have a few continuous features,...