Questions tagged [data-imputation]

Data imputation is the process of replacing missing data with substituted values. This could involve statistically representative data filling (e.g. local averages) or simply replacing the missing data with encoded values (e.g. replace NaNs with zeros).

Filter by
Sorted by
Tagged with
0 votes
2 answers
33 views

How to evaluate data imputation techniques

I have a data set with 29 features 8 if them have missing values. I've tried Sklearn simple imputer and all it's strategies KNN imputer and several Number of K Iterative imputer and all combinations ...
user avatar
0 votes
1 answer
36 views

How to compare between two methods of multivariate to filling NA

In the Titanic dataset, I performed two methods to fill Age NA. The first one is regression using Lasso: ...
user avatar
0 votes
0 answers
14 views

Should I inpute the missing values before the train-validation split?

validation is suppose to provide an unbiased evaluation of a model fit on the training data. In that case inputation before the training-validation split could cause an indirect data leakage because ...
user avatar
0 votes
0 answers
31 views

Is SVM a good choice for this imputing a numerical variable?

Let's say I have 10,000 training points, 100,000,000 points to impute, and 5-10 prediction variables/parameters, all numeric (for now). The target variable is numeric, skewed normal with outliers. I ...
user avatar
0 votes
0 answers
12 views

Time Series Data Missing Value Treatment

I have an hourly time series data for a solar plant which covers 3 years (2019, 2020, 2021). I have a categorical feature named WWCode which has 54 unique values. WWCode is actually a weather ...
user avatar
0 votes
1 answer
25 views

Handling missing values in medical data

I have a medical dataset that contains maternal and foetal data during pregnancy. There are some missing values in the dataset that I am unsure how to handle. Here is a short example of my dataset: <...
user avatar
  • 335
0 votes
2 answers
904 views

Merge two dataframes on multiple columns, only if not NaN

Given two Pandas dataframes, how can I use the second dataframe to fill in missing values, given multiple key columns? ...
user avatar
1 vote
0 answers
10 views

Comparing two models with different (naive) baseline

I would like to compare a model with listwise deletion to a model with multiple imputation. However, the model with listwise deletion has a majority class of 70%, while the model with multiple ...
user avatar
  • 11
0 votes
0 answers
10 views

How to use K-NN imputer without replacing with decimal values example ( 0.75,0.6) instead of binary outcome (yes or no, 1 or 0)?

I am trying to impute some missing categorical values using K-NN imputer, after imputation the missing values are replaced with some decimal numbers. I want to use K-NN as classifier and the output (...
user avatar
1 vote
0 answers
61 views

How does Kalman filtering work backwards?

For time series missing value imputation I am using the Kalman filter/smoothing approach, given in the imputeTS package. As Kalman filter is iterative and needs a view data points to make its estimate ...
user avatar
0 votes
2 answers
78 views

Can I impute with median if median = 0?

I want to impute a numerical feature using the median, but the median for that feature is 0 and mean = 106. Should I go ahead and impute or is there anything else I can do? PS: I don't want to create ...
user avatar
  • 1,263
2 votes
0 answers
252 views

Identify MCAR, MNAR and MAR in the data

If I have missing values in a dataset, I can't just blindly impute them with mean/median/mode or any other technique. I have to identify what kind of missing values they are, namely: MCAR (missing ...
user avatar
  • 1,263
2 votes
1 answer
22 views

Imputing Data that Isn't Missing

I have two columns, [Date Activated] and [Date Closed]. One is the date an account was activated, and the other column is the date an account is closed. There are three scenarios: Case 1 (1/6 data) ...
user avatar
0 votes
0 answers
43 views

Comparing imputation strategies on a data set with no missing value

I am trying to evaluate a few data imputation methods and I am trying to see which one has a more accurate result. I have a dataset with no missing values(called X)....
user avatar
  • 101
0 votes
0 answers
14 views

Imputing missing data of an univariate Sample

I have given a set of values $$ (x_1,...,x_n) $$ these are results of a specified model that predicts defaults for credit risks. For some of the observations the in the model needed information is ...
user avatar
0 votes
0 answers
31 views

Categorical column with many missing values (33%)

I have a dataset of credit cards with these columns as feature (20000 rows): ...
user avatar
  • 1
3 votes
1 answer
89 views

Can data leakage be sometimes acceptable?

I have recently started using kaggle and I have stumbled on a few examples of practices I would consider do be data leakage. Many of them were done by people well established on the platform and I ...
user avatar
  • 115
3 votes
4 answers
183 views

Dropping columns or inputing numbers

After looking at the various different ways of inputting data to replace NaN in a dataset vs. dropping observations or columns based on a threshold, the right ...
user avatar
1 vote
2 answers
4k views

Check if a particular value is a datetime and assigning a particular column value in pandas

I have a pandas data frame that contains a partially corrupted data field as below. It has numbers (which are not a date) or nans. The real data frame has an incredibly large number of rows as well. I ...
user avatar
1 vote
1 answer
830 views

XGBoost - Imputing Vs keeping NaN

What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at ...
user avatar
0 votes
2 answers
618 views

Using scikit-learn iterative imputer with extra tree regressor eats a lot of RAM

I'm imputing a table around 150K by 60 floats and has about 45% missing values, I'm using ExtraTreeRegressor with IterativeImputer ...
user avatar
0 votes
1 answer
26 views

Impute missing value: transpose or not?

I'm building a model that fills the missing values from a Dataframe that contains the number of visitors for different stores, each day: day store_a store_b store_c 2021-01-01 100 200 300 2021-01-...
user avatar
2 votes
1 answer
271 views

kNN for non-ordinal variables

kNN is a distance-based method, so it requires the input to be in numerical form. I was wondering if it is possible to use kNN imputer for non-ordinal categorical variables (like color). Since the ...
user avatar
1 vote
1 answer
88 views

Is there potentially data leakage during imputation for time-varying sensor data?

I have a time-varying dataset that contains some missing data. I have sensors that continuously monitor some properties at evenly-spaced intervals and I would like to impute the missing values using ...
user avatar
  • 165
1 vote
1 answer
401 views

How to impute missing value in Test Set using a custom Imputer created on training dataset

I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique. Under this technique, I replaced missing values with ...
user avatar
  • 161
0 votes
0 answers
266 views

Impute missing values in feature column on the basis of Target column

I am working on a toy project for insurance claim prediction. In the input data for one of the feature (numeric data type) half of the values are missing. My target variable is binary which indicates ...
user avatar
  • 161
0 votes
1 answer
417 views

K-Fold cross validation and data leakage

I want to do K-Fold cross validation and also I want to do normalization or feature scaling for each fold. So let's say we have k folds. At each step we take one fold as validation set and the ...
user avatar
1 vote
1 answer
17 views

Dropping Missing Observations under MAR Assumption

Some of the outcome data in my data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing and ...
user avatar
  • 11
1 vote
0 answers
33 views

How to handle large systematic missing data in time series?

I have this time series, where on the weekends, the dependent variable values are missing. It's only a time series, I do not have any exogenous regressors/features. The dependent variable value is an ...
user avatar
0 votes
1 answer
152 views

sklearn KNN imputation

Can I use sklearn's KNN imputer to fit the model to my training set and impute missing values in the test set using the neighbours from training set ? Is it allowed ? Or , Should I only fit and ...
user avatar
2 votes
2 answers
132 views

Missing value Imputation in dataset

I have two separate files for Testing and Training. In the training data, I am dropping rows that contain too many missing values . But , In the test data , I cannot afford to drop the rows so I have ...
user avatar
1 vote
2 answers
1k views

sklearn - How to create a sequential pipeline

Update: The examples in this post were updated I am reposting this question here after not getting a clear answer in a previous SO post I am looking for a help building a data preprocessing pipleline ...
user avatar
1 vote
1 answer
766 views

Using sklearn knn imputation on a large dataset

I have a large dataset ~ 1 million rows by 400 features and I want to impute the missing values using sklearn KNNImputer. Trying this off the bat I hit memory problems, but I think I can solve this by ...
user avatar
0 votes
0 answers
24 views

If a single element is missing more than 50% of its feature values, should you just remove it?

I have a simple supervised machine learning problem. My training matrix is MxN, where M is the number of records and N is the number of features. I have 600,000 complete patient records and 300,000 ...
user avatar
  • 165
1 vote
1 answer
46 views

In supervised learning, are more data entries always better?

I am doing a supervised learning problem and have 600,000 rows of data. I divided it into a training and test set and achieved a high accuracy that I was happy with. However, I had thrown away 300,000 ...
user avatar
  • 165
0 votes
1 answer
46 views

How to handle sparsely coded features in a dataframe

I have a dataset that contains information regarding diabetes patients, like so: ...
user avatar
  • 335
1 vote
2 answers
503 views

KNN Imputation utilize mean or mode?

In my current project, I am doing KNN imputation with K = 5 and I am using sklearn.impute.KNNImputer. I have a mix of continuous and nominal variables(encoded as 0/1 or ordinal ones that have been ...
user avatar
  • 63
1 vote
1 answer
61 views

How can we use mean imputation without violating feature correlation?

Mean imputation is generally bad practice because it doesn’t take into account feature correlation. Imagine we have a table showing age and fitness score and imagine that an eighty-year-old has a ...
user avatar
1 vote
1 answer
99 views

Dealing with missing data

I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both ...
user avatar
2 votes
1 answer
1k views

How to impute using simple imputer (custom function)

I am imputing my data using simple imputer from sklearn. i want to test many different ways of applying transformations to the data. i.e for logisitcic regression i would like to remove nans and ...
user avatar
  • 466
2 votes
4 answers
322 views

Handling categorical missing values ML

I have gone through this regarding handling missing values in categorical data. Dataset has about 6 categorical columns with ...
user avatar
  • 153
1 vote
2 answers
909 views

What is the point of using MissingIndicator in Scikit-learn?

I have recently discovered Sklearn's MissingIndicator but still wondering how could it improve the usual machine learning work. Clear that ...
user avatar
  • 907
1 vote
0 answers
18 views

Fit model function out defined data range

I have asked this on SO but it has not been well accepted because it seems to be more about data science than programming. Let's say I have a set of data ...
user avatar
  • 111
1 vote
2 answers
252 views

Right order for Data preparation in Machine Learning

For the below mentioned steps of data preparation Outlier detection/treatment Data imputation Data scaling/standardisation Class balancing There are two sub questions Should each of these steps ...
user avatar
11 votes
5 answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
user avatar
0 votes
1 answer
164 views

How to impute right-censored data

I have a dataset of vectors representing movement with various characteristics. Some vectors represents the movement that was stopped by external factor and therefore, observed value for length of ...
user avatar
  • 95
-1 votes
1 answer
365 views

How to handle time series missing values

I have a database of thermal consumption of 100 buildings. Each file has two columns, one is timestamp and the other is usage. My task is to build a prediction model for forecasting the usage for the ...
user avatar
2 votes
2 answers
1k views

What do I do when my column has 50% data missing?

I have a data set i am pre-processing. However in my categorical columns (3 of them) i have "??" in it's place. They constitute 50% of the data. In fact 3 columns have this. My question is how should ...
user avatar
  • 466
1 vote
1 answer
24 views

Dealing with issues in "test" predictons for single "items" (null values, standardization in place, etc)

I know this is kind of a broad question but I have tried to scour both this forum and the internet in general to no avail for this particular situation. So imagine I have a model trained for which, ...
user avatar
  • 159
1 vote
0 answers
69 views

What is the difference between "fit_transform" and "transform" methods when using "SimpleImputer"? [duplicate]

I have following code, I am not able to understand the difference between use of fit_transform() and transform() method in this ...
user avatar