Questions tagged [data-imputation]

Data imputation is the process of replacing missing data with substituted values. This could involve statistically representative data filling (e.g. local averages) or simply replacing the missing data with encoded values (e.g. replace NaNs with zeros).

24 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
2answers
53 views

Missing value Imputation in dataset

I have two separate files for Testing and Training. In the training data, I am dropping rows that contain too many missing values . But , In the test data , I cannot afford to drop the rows so I have ...
3
votes
1answer
1k views

Advice on imputing temperature data with StatsModels MICE

This may be a dumb question but I can't figure out how to actually get the values imputed using StatsModels MICE back into my data. I have a dataframe (dfLocal) with hourly temperature records for ...
2
votes
1answer
57 views

kNN for non-ordinal variables

kNN is a distance-based method, so it requires the input to be in numerical form. I was wondering if it is possible to use kNN imputer for non-ordinal categorical variables (like color). Since the ...
2
votes
1answer
74 views

How to implement single Imputation from conditional distribution?

In [*] page 264, a method of drawing a missing value from a conditional distribution $P(\bf{x}_{mis}|\bf{x}_{obs};\theta)$ which is defined as: I did not find any code implementation of this ...
2
votes
1answer
245 views

Missing population values in census data

I have population data from Census.gov: Total US population by age by year from 1940 through 2010 Depending on the range of decades, the data is missing discrete population values for ages greater ...
1
vote
1answer
14 views

Dropping Missing Observations under MAR Assumption

Some of the outcome data in my data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing and ...
1
vote
0answers
30 views

How to handle large systematic missing data in time series?

I have this time series, where on the weekends, the dependent variable values are missing. It's only a time series, I do not have any exogenous regressors/features. The dependent variable value is an ...
1
vote
1answer
328 views

Using sklearn knn imputation on a large dataset

I have a large dataset ~ 1 million rows by 400 features and I want to impute the missing values using sklearn KNNImputer. Trying this off the bat I hit memory problems, but I think I can solve this by ...
1
vote
2answers
276 views

KNN Imputation utilize mean or mode?

In my current project, I am doing KNN imputation with K = 5 and I am using sklearn.impute.KNNImputer. I have a mix of continuous and nominal variables(encoded as 0/1 or ordinal ones that have been ...
1
vote
1answer
719 views

How to impute using simple imputer (custom function)

I am imputing my data using simple imputer from sklearn. i want to test many different ways of applying transformations to the data. i.e for logisitcic regression i would like to remove nans and ...
1
vote
0answers
17 views

Fit model function out defined data range

I have asked this on SO but it has not been well accepted because it seems to be more about data science than programming. Let's say I have a set of data ...
1
vote
2answers
107 views

Right order for Data preparation in Machine Learning

For the below mentioned steps of data preparation Outlier detection/treatment Data imputation Data scaling/standardisation Class balancing There are two sub questions Should each of these steps ...
1
vote
0answers
137 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
1
vote
1answer
556 views

Is there are way to impute missing values by clustering, regression and stochastic regression

I'd like to know if there are any libraries that allow imputation by clustering, regression and stochastic regression. So far, I've done imputation by mean, median and KNN. I'm trying to evaluate the ...
0
votes
0answers
11 views

Linear models: Imputing missing not at random

This question is a continuation of a similar question for linear models instead of Tree-based model. Given that linear models (e.g. lasso, ridge, Linear regression, elastic net, etc.) can't handle ...
0
votes
0answers
23 views

How to fill missing latitude and longitude values in time series data?

I have time-series data like this: date longitude latitude 01/01/2010 -5.42766 107.5784 02/01/2010 -6.42728 104.5245 07/01/2010 -7.42702 105.5816 14/01/2010 -4.42728 99.57834 17/01/2010 -6.41523 ...
0
votes
1answer
17 views

Impute missing value: transpose or not?

I'm building a model that fills the missing values from a Dataframe that contains the number of visitors for different stores, each day: day store_a store_b store_c 2021-01-01 100 200 300 2021-01-...
0
votes
0answers
10 views

How to impute when you have multiple variables not important to the column with the missing values?

I have a dataset with 18000 rows and 192 columns. I have a specific column with more than 2000 rows missing. I've tried some types of imputations but they just take to long or just seem not good ...
0
votes
0answers
31 views

How can I compare imputation techniques on a dataset with sci-kit learn?

I have a dataset data that has missing values. I am trying two ways of imputing these values, but I would like to compare them. In the first method I am using a ...
0
votes
0answers
89 views

Impute missing values in feature column on the basis of Target column

I am working on a toy project for insurance claim prediction. In the input data for one of the feature (numeric data type) half of the values are missing. My target variable is binary which indicates ...
0
votes
0answers
111 views

sklearn SimpleImputer using mean from different column groups?

I'm looking at the SimpleImputer, in particular in here, and I would like to do the imputation on different columns. My data has 3 different sample groups, and I would like to do ...
0
votes
0answers
23 views

If a single element is missing more than 50% of its feature values, should you just remove it?

I have a simple supervised machine learning problem. My training matrix is MxN, where M is the number of records and N is the number of features. I have 600,000 complete patient records and 300,000 ...
0
votes
1answer
46 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...
-1
votes
1answer
311 views

How to handle time series missing values

I have a database of thermal consumption of 100 buildings. Each file has two columns, one is timestamp and the other is usage. My task is to build a prediction model for forecasting the usage for the ...