Questions tagged [data-imputation]
Data imputation is the process of replacing missing data with substituted values. This could involve statistically representative data filling (e.g. local averages) or simply replacing the missing data with encoded values (e.g. replace NaNs with zeros).
116
questions
1
vote
2
answers
22
views
Does it make sense to impute the target variable when there are a few missing target values in my dataset?
I'm relatively new to machine learning and just trying to have a solid understanding of the basics.
I scraped a real estate/house prices dataset off a website, which I am in the process of cleaning ...
- 11
7
votes
5
answers
443
views
How to handle missing value if imputation doesnt make sense
I have column/feature in my dataset showing years a person has been married "years_married".
Since not every person is married there are NaN fields.
It does not make sense to fillna(0) "...
- 111
0
votes
0
answers
19
views
Python Impute using BayesianRidge() sklearn impute.IterativeImputer regression impute analysis value error
PROBLEM
Use interativeImputer from sklearn.impute.IterativeImputer, to get regression model fit for for BayesianRidge() for impute missing data in variable 'Frontage'.
After the interative_imputer_fit ...
0
votes
0
answers
18
views
Imputing Survival Analysis Data in R
The dataset contains 418 observations and 18 total variables. The two dependent variables are N_Days and Status. The dependent ...
- 3
1
vote
2
answers
46
views
Training a Model that Doesn't Always Have All the Features
I am creating a model that gathers data from multiple sources and determines a confidence level for an instance that is common across those sources (ie. all sources have different features, but all ...
- 13
0
votes
1
answer
86
views
Trying to fill null values with sub-grouped mean value using pandas fillna() and groupby().transform() is doing nothing with the null values
I'm doing EDA on the US police shooting dataset where I'm stuck with filling the 482 null values for the age column. Since no strategy was mentioned on how to fill ...
- 101
0
votes
0
answers
26
views
Minimum strategy for SimpleImputer of scikit-learn
I'm predicting Boston housing prices. There are missing values in the dataset, for example the year in which the house was built. It makes sense to replace these nan...
- 101
1
vote
1
answer
19
views
Imputing missing values for "days ago" feature
I have a dataset with features such as last_visit_n_days_ago, last_purchase_n_days_ago. These features are unavailable for many ...
0
votes
1
answer
23
views
What should you do with NaN Values?
I have a dataset with a number of NaN values in it. I believe ~13,000 rows are affected out of ~500,000, so about 2.6% of the dataset.
I know that I can remove ...
- 523
0
votes
0
answers
13
views
Imputing values for age of property
I have a dataset with property values for a variety of different types of property. I have the age (from build year) of each property however some of the properties are simply "undeveloped ...
0
votes
0
answers
15
views
Missing value imputation with text input
I'm trying to convert product shorthands into brand, pack sizes etc.
E.g. input: "CG EXHIBITIONIST VERY BLACK WP MASCARA", should produce the output Brand = "Cover Girl", Sub-brand ...
- 1
0
votes
1
answer
30
views
Variable selection and NA
I have a very large dataset with a lot of NAs in the data. I want to perform an analysis and have to select the variables that are of most interest.
I feel like I have to take 3 steps before I can ...
- 13
0
votes
0
answers
12
views
how OpenCV RandomForest handles NaN on inference?
I have a trained model of RF regressor using openCV and I want to know how it handles missing values on inference.
I know that some RF implementations do not handle missing values (e.g. scikit-learn), ...
- 1
0
votes
0
answers
24
views
While predicting on test dataset using RandomForestClassifier, I get error, "Input contains NaN, infinity or a value too large for dtype('float32')."
After seeing many recommendations on StackOverflow and also here, I did the train test split first and then imputation.
...
- 1
0
votes
0
answers
19
views
What method of imputation would you choose for a distribution like this
From the following image, it is obvious that while the values start to converge after a while, close to zero they are unstable.
What method of imputation would you choose for a distribution like this, ...
- 518
3
votes
1
answer
84
views
What problems could arise from imputing missing values with impossible values?
I have a dataset that has a lot of missing values that it's not viable to drop the rows and it's also not viable to impute with mean/median since it's quite a significant portion of the data and I don'...
- 31
1
vote
1
answer
70
views
Handling of readability scores for short texts
I have a classification problem using emails as my dataset.
I would like to use scores from various readability formulas as features for the classification. However, most of them are defined for ...
- 518
3
votes
2
answers
101
views
Imputing time data for an event that hasn't occurred yet
Suppose you are trying to predict if lightning will strike in a given location in a given month. (Please ignore the meteorology here, this is not my actual problem, just a hypothetical instance of the ...
- 31
0
votes
2
answers
198
views
Should I deal with missing values first then transform the data or vice versa?
I am currently working on a project involving time series banking stock price data. I have around 3000 observations, some columns have a lot of missing values (null value); they can account for 5 to ...
0
votes
1
answer
56
views
Is it possible to implement logistic regression (or any other ML method) to impute null values in a categorical feature with multiple values?
I'm doing a Data Science project, and I'm on the stage of cleaning categorical features. I've been researching, and it seems that imputing the mean or median can change the distribution. Therefore, a ...
- 25
0
votes
2
answers
168
views
How to evaluate data imputation techniques
I have a data set with 29 features 8 if them have missing values.
I've tried Sklearn simple imputer and all it's strategies
KNN imputer and several Number of K
Iterative imputer and all combinations ...
0
votes
1
answer
42
views
How to compare between two methods of multivariate to filling NA
In the Titanic dataset, I performed two methods to fill Age NA.
The first one is regression using Lasso:
...
0
votes
0
answers
17
views
Should I inpute the missing values before the train-validation split?
validation is suppose to provide an unbiased evaluation of a model fit on the training data. In that case inputation before the training-validation split could cause an indirect data leakage because ...
0
votes
0
answers
33
views
Is SVM a good choice for this imputing a numerical variable?
Let's say I have 10,000 training points, 100,000,000 points to impute, and 5-10 prediction variables/parameters, all numeric (for now). The target variable is numeric, skewed normal with outliers. I ...
- 1
0
votes
1
answer
42
views
Handling missing values in medical data
I have a medical dataset that contains maternal and foetal data during pregnancy. There are some missing values in the dataset that I am unsure how to handle.
Here is a short example of my dataset:
<...
- 365
0
votes
2
answers
3k
views
Merge two dataframes on multiple columns, only if not NaN
Given two Pandas dataframes, how can I use the second dataframe to fill in missing values, given multiple key columns?
...
- 130
1
vote
0
answers
13
views
Comparing two models with different (naive) baseline
I would like to compare a model with listwise deletion to a model with multiple imputation. However, the model with listwise deletion has a majority class of 70%, while the model with multiple ...
- 11
1
vote
0
answers
136
views
How does Kalman filtering work backwards?
For time series missing value imputation I am using the Kalman filter/smoothing approach, given in the imputeTS package.
As Kalman filter is iterative and needs a view data points to make its estimate ...
- 145
0
votes
2
answers
112
views
Can I impute with median if median = 0?
I want to impute a numerical feature using the median, but the median for that feature is 0 and mean = 106. Should I go ahead and impute or is there anything else I can do?
PS: I don't want to create ...
- 1,321
2
votes
0
answers
818
views
Identify MCAR, MNAR and MAR in the data
If I have missing values in a dataset, I can't just blindly impute them with mean/median/mode or any other technique. I have to identify what kind of missing values they are, namely:
MCAR (missing ...
- 1,321
2
votes
1
answer
27
views
Imputing Data that Isn't Missing
I have two columns, [Date Activated] and [Date Closed].
One is the date an account was activated, and the other column is the date an account is closed.
There are three scenarios:
Case 1 (1/6 data) ...
- 21
3
votes
1
answer
111
views
Can data leakage be sometimes acceptable?
I have recently started using kaggle and I have stumbled on a few examples of practices I would consider do be data leakage. Many of them were done by people well established on the platform and I ...
- 115
3
votes
4
answers
208
views
Dropping columns or inputing numbers
After looking at the various different ways of inputting data to replace NaN in a dataset vs. dropping observations or columns based on a threshold, the right ...
- 113
1
vote
2
answers
7k
views
Check if a particular value is a datetime and assigning a particular column value in pandas
I have a pandas data frame that contains a partially corrupted data field as below. It has numbers (which are not a date) or nans. The real data frame has an incredibly large number of rows as well. I ...
4
votes
1
answer
2k
views
XGBoost - Imputing Vs keeping NaN
What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at ...
- 683
0
votes
2
answers
887
views
Using scikit-learn iterative imputer with extra tree regressor eats a lot of RAM
I'm imputing a table around 150K by 60 floats and has about 45% missing values, I'm using ExtraTreeRegressor with IterativeImputer
...
- 53
0
votes
1
answer
36
views
Impute missing value: transpose or not?
I'm building a model that fills the missing values from a Dataframe that contains the number of visitors for different stores, each day:
day
store_a
store_b
store_c
2021-01-01
100
200
300
2021-01-...
- 103
2
votes
1
answer
442
views
kNN for non-ordinal variables
kNN is a distance-based method, so it requires the input to be in numerical form.
I was wondering if it is possible to use kNN imputer for non-ordinal categorical variables (like color).
Since the ...
1
vote
1
answer
188
views
Is there potentially data leakage during imputation for time-varying sensor data?
I have a time-varying dataset that contains some missing data. I have sensors that continuously monitor some properties at evenly-spaced intervals and I would like to impute the missing values using ...
- 185
1
vote
1
answer
553
views
How to impute missing value in Test Set using a custom Imputer created on training dataset
I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique.
Under this technique, I replaced missing values with ...
- 173
0
votes
0
answers
438
views
Impute missing values in feature column on the basis of Target column
I am working on a toy project for insurance claim prediction. In the input data for one of the feature (numeric data type) half of the values are missing. My target variable is binary which indicates ...
- 173
1
vote
1
answer
779
views
K-Fold cross validation and data leakage
I want to do K-Fold cross validation and also I want to do normalization or feature scaling for each fold. So let's say we have k folds. At each step we take one fold as validation set and the ...
user109341
1
vote
1
answer
23
views
Dropping Missing Observations under MAR Assumption
Some of the outcome data in my data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing and ...
- 11
1
vote
0
answers
35
views
How to handle large systematic missing data in time series?
I have this time series, where on the weekends, the dependent variable values are missing.
It's only a time series, I do not have any exogenous regressors/features. The dependent variable value is an ...
- 11
0
votes
1
answer
210
views
sklearn KNN imputation
Can I use sklearn's KNN imputer to fit the model to my training set and impute missing values in the test set using the neighbours from training set ?
Is it allowed ? Or , Should I only fit and ...
- 215
2
votes
2
answers
185
views
Missing value Imputation in dataset
I have two separate files for Testing and Training.
In the training data, I am dropping rows that contain too many missing values .
But , In the test data , I cannot afford to drop the rows so I have ...
- 75
1
vote
2
answers
2k
views
sklearn - How to create a sequential pipeline
Update: The examples in this post were updated
I am reposting this question here after not getting a clear answer in a previous SO post
I am looking for a help building a data preprocessing pipleline ...
- 683
2
votes
1
answer
1k
views
Using sklearn knn imputation on a large dataset
I have a large dataset ~ 1 million rows by 400 features and I want to impute the missing values using sklearn KNNImputer.
Trying this off the bat I hit memory problems, but I think I can solve this by ...
- 121
0
votes
0
answers
24
views
If a single element is missing more than 50% of its feature values, should you just remove it?
I have a simple supervised machine learning problem. My training matrix is MxN, where M is the number of records and N is the number of features. I have 600,000 complete patient records and 300,000 ...
- 185
1
vote
1
answer
47
views
In supervised learning, are more data entries always better?
I am doing a supervised learning problem and have 600,000 rows of data. I divided it into a training and test set and achieved a high accuracy that I was happy with. However, I had thrown away 300,000 ...
- 185