Questions tagged [data-imputation]
Data imputation is the process of replacing missing data with substituted values. This could involve statistically representative data filling (e.g. local averages) or simply replacing the missing data with encoded values (e.g. replace NaNs with zeros).
129 questions
0
votes
0
answers
23
views
Is using KNN Imputation Method make sense in handling missing values for a wind forecast datasets?
I have a datasets called full_weather.csv with various features. I have dropped some columns that does not correlate with wind forecasting and I have found a lots of NaN ( null values in some of the ...
- 11
0
votes
0
answers
10
views
Would imputing using the target variable then analysing correlation between variables be bad due to bias
I have mortality and nutritional data for countries, the mortality data is full for every year but the nutritional data is very limited maybe 2 or 3 years of nutritional data within a 40 year period ...
- 1
0
votes
0
answers
24
views
A question on kNN imputation
I would like to build a machine learning model given a (tabular) dataset. A way to deal with missing values which are assumed to be missing at random is the kNN algorithm. My question pertains to the ...
0
votes
1
answer
30
views
Filling a lot of missing values with arbitrary value
I have a dataset of say 1 million observations. As a silly example, say we want to predict if a person can become a data scientist or not (0/1). I have variables that have a lot of missing values but ...
- 101
0
votes
0
answers
18
views
Geospatial-temporal data interpolation and imputation
I have the dataset that is list of longititudes, latitudes and timestamps. This dataset is representation of vehicle trip. Data can has missing values and some noise. I want to know what methods ...
1
vote
0
answers
24
views
2
votes
0
answers
16
views
Should Imputation Models be Cross Validated
I have a project where I am predicting the best schools based on a series of tests scores, teacher attendance rates, etc. I would like to predict the best school to go to. Some of the data is of ...
- 133
0
votes
0
answers
15
views
Data Imputation and Cluster Analysis
Is there a preferred method of imputation for cluster analysis? Would the selection of imputation method be different if it were dbscan versus hierarchical clustering?
It seems there is an ongoing ...
- 133
0
votes
1
answer
33
views
handling predictions with optional or missing features
We have a few variables that are highly predictive in our modeling task. Is it sound to train models with a superset of features even though some are known NOT to be available at predict time? & ...
- 351
1
vote
1
answer
31
views
Change of data shape when using IterativeImputer from sklearn
I am using the IterativeImputer from sklearn and I notice that it changes the data shape. Initially I have an (X,5) array where all columns except for the last one contain the missing value (which has ...
- 111
2
votes
0
answers
51
views
Best practices for handling "NA" when all NA values exist due to being below the limit of detection?
I am working in R, and have a data set which has a few metabolite concentration values as continuous variables. Anywhere that the concentration was too low to be detected it simply says <LOD. This ...
- 21
1
vote
0
answers
55
views
Imputing techniques for missing values
Currently, I am working on a project for which I would appreciate some feedback and opinions.
My dataset contains data about daily solar radiation from 8 stations and covers the period from 2017 to ...
- 111
1
vote
0
answers
61
views
How do I know the appropriate number of iterations when using Miceforest for imputation?
I want to know how to avoid overfitting without having to increase the number of iterations excessively in Python with the Miceforest library. I know you can make a correlation map of data sets but I ...
3
votes
0
answers
140
views
Why is multiple imputation not used more widely in Data Science?
I have a background in statistics. Multiple imputation is very commonly used to handle missing data, and if it is not used it almost always results in serious criticism. Recently I have been ...
- 141
0
votes
1
answer
443
views
Python SK-Learn KNN Imputer ( "ValueError: could not convert string to float: )
I have data with missing values.
All columns are integer, except for a column that has missing values.
These missing values, were set with a "?" which was converted to NaN using the Numpy ...
- 103
1
vote
1
answer
34
views
How to impute and aggregate data with ID variant variables for predictive modeling?
I have a dataset that looks like so
ID
var_id_invariant_1
...
var_id_invariant_p
var_id_variant_1
...
var_id_variant_k
target
315
25
...
a
2.4
...
A
1
246
31
...
nan
5.7
...
B
0
315
25
...
a
9.4
.....
- 135
0
votes
1
answer
25
views
Interpolating missing data given county and state totals
Problem:
I have population data for states and their constituent counties over several years. Each row is uniquely identified by a state/county and a year. There are four population columns: ...
1
vote
1
answer
753
views
Imputation in train or test data
I'm having a rather simple question.
Let's say i want to do a median imputation. I've read in some places that you should do:
...
0
votes
1
answer
201
views
How can I compare the accuracy of imputation models if there is already missing dataset in the file?
Let's say I have a dataset of 50,000 where about 2% were already missing from the beginning. From what I have learned, we need to use indicators to compare the imputation model with the ground truth ...
1
vote
1
answer
94
views
What is the benefit of using imputation to handle missing values in training data?
This is a bit of a theoretical question. Does imputation to handle missing values in training data add that much benefit to the final prediction accuracy? It seems to me that the imputation would be ...
- 111
0
votes
2
answers
50
views
Data imputation for heavily missing features
I am currently working on the dataset IEEE-CIS Fraud Detection, provided via Kaggle, with around 350 features, with around 600k instances. However, some features are missing large amounts of values, ...
2
votes
1
answer
223
views
Does it make sense to impute the target variable when there are a few missing target values in my dataset?
I'm relatively new to machine learning and just trying to have a solid understanding of the basics.
I scraped a real estate/house prices dataset off a website, which I am in the process of cleaning ...
- 21
7
votes
5
answers
936
views
How to handle missing value if imputation doesnt make sense
I have column/feature in my dataset showing years a person has been married "years_married".
Since not every person is married there are NaN fields.
It does not make sense to fillna(0) "...
- 131
1
vote
2
answers
1k
views
Training a Model that Doesn't Always Have All the Features
I am creating a model that gathers data from multiple sources and determines a confidence level for an instance that is common across those sources (ie. all sources have different features, but all ...
- 13
1
vote
1
answer
2k
views
Trying to fill null values with sub-grouped mean value using pandas fillna() and groupby().transform() is doing nothing with the null values
I'm doing EDA on the US police shooting dataset where I'm stuck with filling the 482 null values for the age column. Since no strategy was mentioned on how to fill ...
- 121
1
vote
1
answer
38
views
Imputing missing values for "days ago" feature
I have a dataset with features such as last_visit_n_days_ago, last_purchase_n_days_ago. These features are unavailable for many ...
0
votes
1
answer
124
views
Handling NaN values in a feature representing prioritization ranking
I have a data set like so
...
- 131
0
votes
1
answer
2k
views
What should you do with NaN Values?
I have a dataset with a number of NaN values in it. I believe ~13,000 rows are affected out of ~500,000, so about 2.6% of the dataset.
I know that I can remove ...
- 671
0
votes
1
answer
177
views
Variable selection and NA
I have a very large dataset with a lot of NAs in the data. I want to perform an analysis and have to select the variables that are of most interest.
I feel like I have to take 3 steps before I can ...
- 13
3
votes
1
answer
126
views
What problems could arise from imputing missing values with impossible values?
I have a dataset that has a lot of missing values that it's not viable to drop the rows and it's also not viable to impute with mean/median since it's quite a significant portion of the data and I don'...
- 31
2
votes
1
answer
455
views
Handling of readability scores for short texts
I have a classification problem using emails as my dataset.
I would like to use scores from various readability formulas as features for the classification. However, most of them are defined for ...
- 636
3
votes
2
answers
205
views
Imputing time data for an event that hasn't occurred yet
Suppose you are trying to predict if lightning will strike in a given location in a given month. (Please ignore the meteorology here, this is not my actual problem, just a hypothetical instance of the ...
- 31
0
votes
3
answers
2k
views
Should I deal with missing values first then transform the data or vice versa?
I am currently working on a project involving time series banking stock price data. I have around 3000 observations, some columns have a lot of missing values (null value); they can account for 5 to ...
0
votes
1
answer
234
views
Is it possible to implement logistic regression (or any other ML method) to impute null values in a categorical feature with multiple values?
I'm doing a Data Science project, and I'm on the stage of cleaning categorical features. I've been researching, and it seems that imputing the mean or median can change the distribution. Therefore, a ...
- 25
0
votes
2
answers
590
views
How to evaluate data imputation techniques
I have a data set with 29 features 8 if them have missing values.
I've tried Sklearn simple imputer and all it's strategies
KNN imputer and several Number of K
Iterative imputer and all combinations ...
0
votes
1
answer
66
views
How to compare between two methods of multivariate to filling NA
In the Titanic dataset, I performed two methods to fill Age NA.
The first one is regression using Lasso:
...
- 11
0
votes
0
answers
35
views
Is SVM a good choice for this imputing a numerical variable?
Let's say I have 10,000 training points, 100,000,000 points to impute, and 5-10 prediction variables/parameters, all numeric (for now). The target variable is numeric, skewed normal with outliers. I ...
- 1
0
votes
1
answer
72
views
Handling missing values in medical data
I have a medical dataset that contains maternal and foetal data during pregnancy. There are some missing values in the dataset that I am unsure how to handle.
Here is a short example of my dataset:
<...
- 447
0
votes
2
answers
7k
views
Merge two dataframes on multiple columns, only if not NaN
Given two Pandas dataframes, how can I use the second dataframe to fill in missing values, given multiple key columns?
...
- 130
1
vote
0
answers
18
views
Comparing two models with different (naive) baseline
I would like to compare a model with listwise deletion to a model with multiple imputation. However, the model with listwise deletion has a majority class of 70%, while the model with multiple ...
- 11
1
vote
0
answers
342
views
How does Kalman filtering work backwards?
For time series missing value imputation I am using the Kalman filter/smoothing approach, given in the imputeTS package.
As Kalman filter is iterative and needs a view data points to make its estimate ...
- 145
0
votes
2
answers
177
views
Can I impute with median if median = 0?
I want to impute a numerical feature using the median, but the median for that feature is 0 and mean = 106. Should I go ahead and impute or is there anything else I can do?
PS: I don't want to create ...
- 2,165
3
votes
0
answers
2k
views
Identify MCAR, MNAR and MAR in the data
If I have missing values in a dataset, I can't just blindly impute them with mean/median/mode or any other technique. I have to identify what kind of missing values they are, namely:
MCAR (missing ...
- 2,165
2
votes
1
answer
38
views
Imputing Data that Isn't Missing
I have two columns, [Date Activated] and [Date Closed].
One is the date an account was activated, and the other column is the date an account is closed.
There are three scenarios:
Case 1 (1/6 data) ...
- 21
3
votes
1
answer
156
views
Can data leakage be sometimes acceptable?
I have recently started using kaggle and I have stumbled on a few examples of practices I would consider do be data leakage. Many of them were done by people well established on the platform and I ...
- 135
3
votes
4
answers
274
views
Dropping columns or inputing numbers
After looking at the various different ways of inputting data to replace NaN in a dataset vs. dropping observations or columns based on a threshold, the right ...
- 111
2
votes
2
answers
13k
views
Check if a particular value is a datetime and assigning a particular column value in pandas
I have a pandas data frame that contains a partially corrupted data field as below. It has numbers (which are not a date) or nans. The real data frame has an incredibly large number of rows as well. I ...
4
votes
1
answer
6k
views
XGBoost - Imputing Vs keeping NaN
What is the benefit of imputing numerical or categorical features when using DT methods such as XGBoost that can handle missing values? This question is mainly for when the values are missing not at ...
- 755
0
votes
2
answers
2k
views
Using scikit-learn iterative imputer with extra tree regressor eats a lot of RAM
I'm imputing a table around 150K by 60 floats and has about 45% missing values, I'm using ExtraTreeRegressor with IterativeImputer
...
- 53
0
votes
1
answer
56
views
Impute missing value: transpose or not?
I'm building a model that fills the missing values from a Dataframe that contains the number of visitors for different stores, each day:
day
store_a
store_b
store_c
2021-01-01
100
200
300
2021-01-...
- 103