Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-imputation]

The tag has no usage guidance.

2
votes
3answers
34 views

Machine Learning, Imputing values that should be blank

Sometimes data sets contain variables that indicate the presence of an event and the value that represented the event. As an example say a teacher wants to predict the grades of his students. Some of ...
1
vote
0answers
49 views

Target Encoding: missing value imputation before or after encoding

I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values. Let's say I have a few continuous features,...
2
votes
1answer
21 views

Training on data with inherently non-applicable data cells

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense. Chemical Dataset Contains thousands of rows of chemical mixtures with many columns of ...
0
votes
1answer
21 views

The proper way to codify Na in a list in R

I am trying to impute missing timeseries present in different dimentions, row by row, on the whole date set. I showed the type of return of na.kalman() and it ...
0
votes
0answers
20 views

Any thoughts on how to fill missing (isolated, and ranges) annual data to improve accuracy for future predictions

The purpose is to have a better training set, it's a multivariate timeseries ranging from 11(November) to 8(August). So 9(September) and 10(October) are totally missing. Here is an extract, a one ...
2
votes
2answers
26 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
1
vote
1answer
107 views

How to handle missing data data in dependent variable?

I'm solving a ML problem statement where there are around 40k records in the dataset. A dependent variable is given in the question (There are many independent variables). But there are some 2k ...
2
votes
2answers
130 views

How to fill missing numeric if any value in a subset is missing, all other columns with the same subset are missing

There is a clear pattern that show for two separate subsets (set of columns); If one value is missing in a column, values of other columns in the same subset are missing for any row. Here is a ...
2
votes
0answers
117 views

Handling missing values to optimize polynomial features

I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on ...
0
votes
1answer
31 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...
0
votes
1answer
263 views

pandas: How to impute the categorical column by the nearest neighbors?

I've a categorical column with values such as right('r'), left('l') and straight('s'). I expect these to have a continuum periods in the data and want to impute nans with the most plausible value in ...
0
votes
1answer
39 views

Missing Values in Classification

I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't. I also have ...
0
votes
1answer
51 views

How to deal with missing data for only some categories

Or in other words, data for category A is irrelevant for category B. So it is not present, how can imputing missing data distort/effect learning models broadly. I can't find any logic how to deal ...
2
votes
2answers
82 views

What is the difference between Missing at Random and Missing not at Random data?

I have been working with a dataset where the missing data seem to following a few particular patterns. I have gone through a lot websites and articles related to missing data but I haven't been able ...
0
votes
2answers
43 views

what to do if the missing data in one column is based on some value/condition in another column in r?

I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever ...
1
vote
1answer
200 views

Imputation missing values other than using Mean, Median in python

I heard that Mean, Median isn't the best way to impute the missing values, why would that be? In my scenario, I have data like this ...
0
votes
2answers
76 views

Basics: What is the correct sequence for preparing simple data for ML?

I'm just getting started with ML and am busy with my first Kaggle competition (the titanic one). I was just wondering what would be the best way to organise the data to avoid redundancy with the ...
1
vote
1answer
47 views

Missing value in continuous variable: Indicator variable vs. Indicator value

Most data has missing values, and as far as I'm aware, these are the options: Imputation (mean, hot-deck, etc.) Indicator variable. A categorical variable that tells what type the primary variable is....
0
votes
2answers
88 views

How to handle NaNs for ratio feature for binary classifier?

I'm creating a churn model and would like to create a ratio (# customers / total transaction) for each merchant. About 70% of the data are NaNs (zero/zero). I was wondering what I should impute for ...
1
vote
2answers
40 views

How would you deal with inf. or NA for rate or ratio as a feature variable

I'm trying to create a feature for a churn model (binary classifier). The feature is mean of sales growth rates for several months. But if I just take the mean of sales for several months, I often ...
1
vote
1answer
301 views

Use of Random Forest algorithm in PySpark for imputation

I am wondering how to use Random Forest algorithm for imputing missing values in a dataset. It is supposed to work well with missing values but I am not sure how those missing values are dealt with ...
1
vote
0answers
350 views

Is there are way to impute missing values by clustering, regression and stochastic regression

I'd like to know if there are any libraries that allow imputation by clustering, regression and stochastic regression. So far, I've done imputation by mean, median and KNN. I'm trying to evaluate the ...
0
votes
1answer
27 views

Percentage of missing values so that we can't perform imputation

(This is not a question on ways to handle missing data) I have a dataset with around $80$ or so features and around $100000$ rows. Several of those features have missing (NULL) values for a "large ...
1
vote
2answers
221 views

How would one impute missing values for a Discrete variable?

How would one imputing missing values (without using the mode) for a discrete variable, e.g. a variable corresponding to a count.
1
vote
1answer
115 views

Missing population values in census data

I have population data from Census.gov: Total US population by age by year from 1940 through 2010 Depending on the range of decades, the data is missing discrete population values for ages greater ...
1
vote
1answer
699 views

Should I Impute target values?

I am new to data science and I am currently playing around a bit. Data exploration and preparation is really annoying. Eventhough I use pandas. I achieved imputing missing values in independant ...
3
votes
2answers
78 views

Define a Model for predicting missing values in Data Set [closed]

I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset. For example: I have a dataset which contains information about a product. This can be ...
1
vote
1answer
205 views

Which is faster for imputing: R or Python? [closed]

I have a fairly large dataset (about 40k rows, 40 columns) with many NA's (up to 40% of each variable). All of my work has been done in R so far, but I know R (which I'm told runs only on a single ...
0
votes
1answer
928 views

Imputing for multiple missing variables using sklearn

I have a dataset of around 10 million rows and around 10 columns. I have missing data that occurs sporadically across 4 of these variables. What technique would you recommend I use in sklearn to fill ...
0
votes
2answers
1k views

Filling missing values with pyspark using a probability distribution

I want to fill missing values in my dataframe. ...
4
votes
2answers
6k views

R's mice imputation alternative in Python

What is Python's alternative to missing data imputation with mice in R? Imputation using median/mean seems pretty lame, I'm looking for other methods of imputation, ...
0
votes
1answer
3k views

Imputation of missing values and dealing with categorical values

I have a dataset (10 million rows, 55 columns) with many missing values. I need to predict those values somehow using other non-missing values, i.e. replace them with something that is not NaN. Mean ...
0
votes
2answers
539 views

how to do the imputation for categorical feature with a missing rate?

I have a dataset containing a categorical feature with a missing rate 95%. What value can replace the missing cells? Or drop this feature?
2
votes
1answer
1k views

Missing data imputation with KNN

-1 down vote favorite I have a dataset including missing data for most of the variables. Assume the dataset is as follows: ...
2
votes
2answers
2k views

Methodologies for predicting missing data

I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset. For example: I have a dataset which contains information about a person. This can be ...
1
vote
3answers
1k views

What would be the best way to impute data?

Other than just filling in with the mean of a feature, what other methods are there which can work well? I am trying to decide whether or not to use a denoising-autoencoder or just impute with the ...
4
votes
2answers
95 views

Check Imputation Efficiency - How To Compare Data Frames?

I try to evaluate several NA imputation methods with supervised approach: I clone my original data frame with no NAs, artifically insert NAs into the resulting Data Frame and apply imputations to the ...
2
votes
1answer
175 views

Missing Categorical Features - no imputation

I've been reading about how to approach missing categorical features in test data, and the most common approach is to use imputation - for example using the last known value or getting the majority ...
1
vote
3answers
118 views

Method for predicting price based on Geographical market, Product, and Company

I have a dataset which tracks the prices of 21 products, charged by 24 companies, in 150 different cities across the globe. However, the data set has missing values--that is, I might have Company X's ...
5
votes
1answer
304 views

Computationally Inexpensive Imputation Techniques in R

I have a large data-frame (155257 x 21 to be specific) with only a few missing values. Say, some 2.16% of the values need to be imputed. The values are floating point numbers. I'd like to use a ...
4
votes
2answers
858 views

How to measure the performance of an imputation technique

I would like to know how I can measure the performance of an imputation technique. I have read a lot about this. Most literature on the web are applying a classifier after the data has been completed. ...
1
vote
1answer
80 views

Using predictive modelling for temperature data set

I am absolutely new to this area of predictive modelling in data science. I am not able to understand how and what modelling techniques do we use? Does it depend on the data type? Does it depend on ...