Questions tagged [data-imputation]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
6
votes
5answers
1k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
0
votes
0answers
15 views

How to impute missing values for hour of day?

I have a dataset containing a certain amount of bookings, no shows and cancellations. We assume that a cancellation less than 2 days is the same as a no show (since you did not have the time to ...
0
votes
0answers
17 views

Input missing data with interpolation [duplicate]

When dealing with missing values there are several strategies that can be used such as: imputing with the mean, median, a fixed value... I have read that you can do things such as interpolation. ...
0
votes
1answer
50 views

How to impute right-censored data

I have a dataset of vectors representing movement with various characteristics. Some vectors represents the movement that was stopped by external factor and therefore, observed value for length of ...
-1
votes
1answer
21 views

How to handle time series missing values

I have a database of thermal consumption of 100 buildings. Each file has two columns, one is timestamp and the other is usage. My task is to build a prediction model for forecasting the usage for the ...
2
votes
2answers
141 views

What do I do when my column has 50% data missing?

I have a data set i am pre-processing. However in my categorical columns (3 of them) i have "??" in it's place. They constitute 50% of the data. In fact 3 columns have this. My question is how should ...
1
vote
1answer
19 views

Dealing with issues in “test” predictons for single “items” (null values, standardization in place, etc)

I know this is kind of a broad question but I have tried to scour both this forum and the internet in general to no avail for this particular situation. So imagine I have a model trained for which, ...
0
votes
0answers
22 views

Memory error while trying to apply Multivariate imputation by chained equations?

I have a training set consisting of movies data, which includes features like runtime, keywords, cast etc. The keywords columns consist of json collections of keywords used in the movie with ...
0
votes
0answers
36 views

What is the difference between “fit_transform” and “transform” methods when using “SimpleImputer”? [duplicate]

I have following code, I am not able to understand the difference between use of fit_transform() and transform() method in this ...
1
vote
0answers
78 views

Retrieve dropped column names from `sklearn.impute.SimpleImputer`

The SimpleImputer class takes pandas dataframes and returns unlabeled numpy arrays. Which means that the SimpleImputer drops ...
2
votes
1answer
51 views

How to implement single Imputation from conditional distribution?

In [*] page 264, a method of drawing a missing value from a conditional distribution $P(\bf{x}_{mis}|\bf{x}_{obs};\theta)$ which is defined as: I did not find any code implementation of this ...
0
votes
0answers
45 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
0
votes
2answers
1k views

How can I replace outliers with maximum non-outlier value?

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no ...
0
votes
0answers
27 views

Model Checking in Multiple Imputation

Trying to understand how to incorporate Multivariate Imputation by Chained Equation (MICE) for handling missing data in python. I know there are a few libraries that can implement MICE on a dataset ...
0
votes
0answers
18 views

What are the similarities and differences between Imputation, Generative Models and Bootstrapping?

What are the similarities and differences between the following methods: Data Imputation, Generative Models, Bootstrapping.
6
votes
2answers
73 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
0
votes
0answers
10 views

Strategy for unrecorded predictors

I'm trying to create a logistic regression model for predicting future admissions based on historic clinical/utilization/demographic information. Although I have three years history available, for ...
0
votes
0answers
9 views

Custom Imputation relative to targets in TRAIN and TEST sets

I have a methodology question for dealing with heaps of missing data in my project. My dataset is composed of parts A (~200 columns) and B (another ~200 columns). Together they are to be used for ...
7
votes
2answers
748 views

Which comes first? Multiple Imputation, Splitting into train/test, or Standardization/Normalization

I am working on a multi-class classification problem, with ~65 features and ~150K instances. 30% of features are categorical and the rest are numerical (continuous). I understand that standardization ...
3
votes
1answer
562 views

Advice on imputing temperature data with StatsModels MICE

This may be a dumb question but I can't figure out how to actually get the values imputed using StatsModels MICE back into my data. I have a dataframe (dfLocal) with hourly temperature records for ...
1
vote
3answers
207 views

Is it right to impute Train and Test set?

I am experimenting with a dataset and I have a couple of columns with high cardinality. So, I performed mean target encoding (given that my dataset had more than 50000 observations). But, before doing ...
4
votes
3answers
496 views

What predictive model to use to impute Gender?

My data looks like this: birth_date has 634,990 missing values gender has 328,849 missing values Both of these are a ...
2
votes
3answers
75 views

Machine Learning, Imputing values that should be blank

Sometimes data sets contain variables that indicate the presence of an event and the value that represented the event. As an example say a teacher wants to predict the grades of his students. Some of ...
3
votes
1answer
240 views

Target Encoding: missing value imputation before or after encoding

I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values. Let's say I have a few continuous features,...
2
votes
1answer
30 views

Training on data with inherently non-applicable data cells

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense. Chemical Dataset Contains thousands of rows of chemical mixtures with many columns of ...
0
votes
1answer
35 views

The proper way to codify Na in a list in R

I am trying to impute missing timeseries present in different dimentions, row by row, on the whole date set. I showed the type of return of na.kalman() and it ...
2
votes
2answers
32 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
1
vote
1answer
1k views

How to handle missing data data in dependent variable?

I'm solving a ML problem statement where there are around 40k records in the dataset. A dependent variable is given in the question (There are many independent variables). But there are some 2k ...
2
votes
2answers
312 views

How to fill missing numeric if any value in a subset is missing, all other columns with the same subset are missing

There is a clear pattern that show for two separate subsets (set of columns); If one value is missing in a column, values of other columns in the same subset are missing for any row. Here is a ...
3
votes
1answer
596 views

Handling missing values to optimize polynomial features

I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on ...
0
votes
1answer
38 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...
0
votes
1answer
573 views

pandas: How to impute the categorical column by the nearest neighbors?

I've a categorical column with values such as right('r'), left('l') and straight('s'). I expect these to have a continuum periods in the data and want to impute nans with the most plausible value in ...
0
votes
1answer
49 views

Missing Values in Classification

I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't. I also have ...
0
votes
1answer
23 views

Investigate why data is missing? After finding out reasons, what should I do next?

...
6
votes
1answer
82 views

How to deal with missing data for only some categories

Or in other words, data for category A is irrelevant for category B. So it is not present, how can imputing missing data distort/effect learning models broadly. I can't find any logic how to deal ...
4
votes
2answers
681 views

What is the difference between Missing at Random and Missing not at Random data?

I have been working with a dataset where the missing data seem to following a few particular patterns. I have gone through a lot websites and articles related to missing data but I haven't been able ...
0
votes
3answers
187 views

what to do if the missing data in one column is based on some value/condition in another column in r?

I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever ...
2
votes
1answer
1k views

Imputation missing values other than using Mean, Median in python

I heard that Mean, Median isn't the best way to impute the missing values, why would that be? In my scenario, I have data like this ...
0
votes
2answers
147 views

Basics: What is the correct sequence for preparing simple data for ML?

I'm just getting started with ML and am busy with my first Kaggle competition (the titanic one). I was just wondering what would be the best way to organise the data to avoid redundancy with the ...
1
vote
1answer
214 views

Missing value in continuous variable: Indicator variable vs. Indicator value

Most data has missing values, and as far as I'm aware, these are the options: Imputation (mean, hot-deck, etc.) Indicator variable. A categorical variable that tells what type the primary variable is....
0
votes
2answers
176 views

How to handle NaNs for ratio feature for binary classifier?

I'm creating a churn model and would like to create a ratio (# customers / total transaction) for each merchant. About 70% of the data are NaNs (zero/zero). I was wondering what I should impute for ...
1
vote
2answers
64 views

How would you deal with inf. or NA for rate or ratio as a feature variable

I'm trying to create a feature for a churn model (binary classifier). The feature is mean of sales growth rates for several months. But if I just take the mean of sales for several months, I often ...
1
vote
1answer
447 views

Use of Random Forest algorithm in PySpark for imputation

I am wondering how to use Random Forest algorithm for imputing missing values in a dataset. It is supposed to work well with missing values but I am not sure how those missing values are dealt with ...
1
vote
0answers
508 views

Is there are way to impute missing values by clustering, regression and stochastic regression

I'd like to know if there are any libraries that allow imputation by clustering, regression and stochastic regression. So far, I've done imputation by mean, median and KNN. I'm trying to evaluate the ...
0
votes
1answer
47 views

Percentage of missing values so that we can't perform imputation

(This is not a question on ways to handle missing data) I have a dataset with around $80$ or so features and around $100000$ rows. Several of those features have missing (NULL) values for a "large ...
1
vote
2answers
497 views

How would one impute missing values for a Discrete variable?

How would one imputing missing values (without using the mode) for a discrete variable, e.g. a variable corresponding to a count.
1
vote
1answer
208 views

Missing population values in census data

I have population data from Census.gov: Total US population by age by year from 1940 through 2010 Depending on the range of decades, the data is missing discrete population values for ages greater ...
2
votes
1answer
1k views

Should I Impute target values?

I am new to data science and I am currently playing around a bit. Data exploration and preparation is really annoying. Eventhough I use pandas. I achieved imputing missing values in independant ...
3
votes
2answers
123 views

Define a Model for predicting missing values in Data Set [closed]

I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset. For example: I have a dataset which contains information about a product. This can be ...
1
vote
1answer
290 views

Which is faster for imputing: R or Python? [closed]

I have a fairly large dataset (about 40k rows, 40 columns) with many NA's (up to 40% of each variable). All of my work has been done in R so far, but I know R (which I'm told runs only on a single ...