Questions tagged [data-imputation]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
15 views

How to handle large systematic missing data in time series?

I have this time series, where on the weekends, the dependent variable values are missing. It's only a time series, I do not have any exogenous regressors/features. The dependent variable value is an ...
0
votes
1answer
21 views

sklearn KNN imputation

Can I use sklearn's KNN imputer to fit the model to my training set and impute missing values in the test set using the neighbours from training set ? Is it allowed ? Or , Should I only fit and ...
1
vote
0answers
13 views

Missing value Imputation in dataset

I have two separate files for Testing and Training. In the training data, I am dropping rows that contain too many missing values . But , In the test data , I cannot afford to drop the rows so I have ...
0
votes
2answers
69 views

sklearn - How to create a sequential pipeline

Update: The examples in this post were updated I am reposting this question here after not getting a clear answer in a previous SO post I am looking for a help building a data preprocessing pipleline ...
0
votes
0answers
8 views

sklearn SimpleImputer using mean from different column groups?

I'm looking at the SimpleImputer, in particular in here, and I would like to do the imputation on different columns. My data has 3 different sample groups, and I would like to do ...
1
vote
1answer
39 views

Using sklearn knn imputation on a large dataset

I have a large dataset ~ 1 million rows by 400 features and I want to impute the missing values using sklearn KNNImputer. Trying this off the bat I hit memory problems, but I think I can solve this by ...
0
votes
0answers
20 views

If a single element is missing more than 50% of its feature values, should you just remove it?

I have a simple supervised machine learning problem. My training matrix is MxN, where M is the number of records and N is the number of features. I have 600,000 complete patient records and 300,000 ...
1
vote
1answer
35 views

In supervised learning, are more data entries always better?

I am doing a supervised learning problem and have 600,000 rows of data. I divided it into a training and test set and achieved a high accuracy that I was happy with. However, I had thrown away 300,000 ...
0
votes
1answer
36 views

How to handle sparsely coded features in a dataframe

I have a dataset that contains information regarding diabetes patients, like so: ...
2
votes
1answer
59 views

KNN Imputation utilize mean or mode?

In my current project, I am doing KNN imputation with K = 5 and I am using sklearn.impute.KNNImputer. I have a mix of continuous and nominal variables(encoded as 0/1 or ordinal ones that have been ...
1
vote
1answer
21 views

How can we use mean imputation without violating feature correlation?

Mean imputation is generally bad practice because it doesn’t take into account feature correlation. Imagine we have a table showing age and fitness score and imagine that an eighty-year-old has a ...
1
vote
1answer
46 views

Dealing with missing data

I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both ...
0
votes
1answer
118 views

How to impute using simple imputer (custom function)

I am imputing my data using simple imputer from sklearn. i want to test many different ways of applying transformations to the data. i.e for logisitcic regression i would like to remove nans and ...
2
votes
3answers
67 views

Handling categorical missing values ML

I have gone through this regarding handling missing values in categorical data. Dataset has about 6 categorical columns with ...
1
vote
2answers
94 views

What is the point of using MissingIndicator in Scikit-learn?

I have recently discovered Sklearn's MissingIndicator but still wondering how could it improve the usual machine learning work. Clear that ...
1
vote
0answers
17 views

Fit model function out defined data range

I have asked this on SO but it has not been well accepted because it seems to be more about data science than programming. Let's say I have a set of data ...
0
votes
2answers
37 views

Right order for Data preparation in Machine Learning

For the below mentioned steps of data preparation Outlier detection/treatment Data imputation Data scaling/standardisation Class balancing There are two sub questions Should each of these steps ...
10
votes
5answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
0
votes
0answers
17 views

How to impute missing values for hour of day?

I have a dataset containing a certain amount of bookings, no shows and cancellations. We assume that a cancellation less than 2 days is the same as a no show (since you did not have the time to ...
0
votes
0answers
28 views

Input missing data with interpolation [duplicate]

When dealing with missing values there are several strategies that can be used such as: imputing with the mean, median, a fixed value... I have read that you can do things such as interpolation. ...
0
votes
1answer
67 views

How to impute right-censored data

I have a dataset of vectors representing movement with various characteristics. Some vectors represents the movement that was stopped by external factor and therefore, observed value for length of ...
-1
votes
1answer
49 views

How to handle time series missing values

I have a database of thermal consumption of 100 buildings. Each file has two columns, one is timestamp and the other is usage. My task is to build a prediction model for forecasting the usage for the ...
2
votes
2answers
220 views

What do I do when my column has 50% data missing?

I have a data set i am pre-processing. However in my categorical columns (3 of them) i have "??" in it's place. They constitute 50% of the data. In fact 3 columns have this. My question is how should ...
1
vote
1answer
19 views

Dealing with issues in “test” predictons for single “items” (null values, standardization in place, etc)

I know this is kind of a broad question but I have tried to scour both this forum and the internet in general to no avail for this particular situation. So imagine I have a model trained for which, ...
0
votes
0answers
30 views

Memory error while trying to apply Multivariate imputation by chained equations?

I have a training set consisting of movies data, which includes features like runtime, keywords, cast etc. The keywords columns consist of json collections of keywords used in the movie with ...
0
votes
0answers
49 views

What is the difference between “fit_transform” and “transform” methods when using “SimpleImputer”? [duplicate]

I have following code, I am not able to understand the difference between use of fit_transform() and transform() method in this ...
4
votes
2answers
364 views

Retrieve dropped column names from `sklearn.impute.SimpleImputer`

The SimpleImputer class takes pandas dataframes and returns unlabeled numpy arrays. Which means that the SimpleImputer drops ...
2
votes
1answer
61 views

How to implement single Imputation from conditional distribution?

In [*] page 264, a method of drawing a missing value from a conditional distribution $P(\bf{x}_{mis}|\bf{x}_{obs};\theta)$ which is defined as: I did not find any code implementation of this ...
0
votes
0answers
86 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
0
votes
2answers
3k views

How can I replace outliers with maximum non-outlier value?

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no ...
6
votes
2answers
139 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
8
votes
2answers
2k views

Which comes first? Multiple Imputation, Splitting into train/test, or Standardization/Normalization

I am working on a multi-class classification problem, with ~65 features and ~150K instances. 30% of features are categorical and the rest are numerical (continuous). I understand that standardization ...
3
votes
1answer
891 views

Advice on imputing temperature data with StatsModels MICE

This may be a dumb question but I can't figure out how to actually get the values imputed using StatsModels MICE back into my data. I have a dataframe (dfLocal) with hourly temperature records for ...
1
vote
3answers
539 views

Is it right to impute Train and Test set?

I am experimenting with a dataset and I have a couple of columns with high cardinality. So, I performed mean target encoding (given that my dataset had more than 50000 observations). But, before doing ...
4
votes
3answers
748 views

What predictive model to use to impute Gender?

My data looks like this: birth_date has 634,990 missing values gender has 328,849 missing values Both of these are a ...
2
votes
3answers
94 views

Machine Learning, Imputing values that should be blank

Sometimes data sets contain variables that indicate the presence of an event and the value that represented the event. As an example say a teacher wants to predict the grades of his students. Some of ...
3
votes
1answer
470 views

Target Encoding: missing value imputation before or after encoding

I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values. Let's say I have a few continuous features,...
2
votes
1answer
36 views

Training on data with inherently non-applicable data cells

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense. Chemical Dataset Contains thousands of rows of chemical mixtures with many columns of ...
0
votes
1answer
47 views

The proper way to codify Na in a list in R

I am trying to impute missing timeseries present in different dimentions, row by row, on the whole date set. I showed the type of return of na.kalman() and it ...
2
votes
2answers
32 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
2
votes
1answer
2k views

How to handle missing data data in dependent variable?

I'm solving a ML problem statement where there are around 40k records in the dataset. A dependent variable is given in the question (There are many independent variables). But there are some 2k ...
2
votes
2answers
343 views

How to fill missing numeric if any value in a subset is missing, all other columns with the same subset are missing

There is a clear pattern that show for two separate subsets (set of columns); If one value is missing in a column, values of other columns in the same subset are missing for any row. Here is a ...
3
votes
1answer
819 views

Handling missing values to optimize polynomial features

I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on ...
0
votes
1answer
43 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...
0
votes
1answer
631 views

pandas: How to impute the categorical column by the nearest neighbors?

I've a categorical column with values such as right('r'), left('l') and straight('s'). I expect these to have a continuum periods in the data and want to impute nans with the most plausible value in ...
0
votes
1answer
49 views

Missing Values in Classification

I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't. I also have ...
0
votes
1answer
24 views

Investigate why data is missing? After finding out reasons, what should I do next?

...
6
votes
1answer
144 views

How to deal with missing data for only some categories

Or in other words, data for category A is irrelevant for category B. So it is not present, how can imputing missing data distort/effect learning models broadly. I can't find any logic how to deal ...
4
votes
2answers
1k views

What is the difference between Missing at Random and Missing not at Random data?

I have been working with a dataset where the missing data seem to following a few particular patterns. I have gone through a lot websites and articles related to missing data but I haven't been able ...
0
votes
3answers
329 views

what to do if the missing data in one column is based on some value/condition in another column in r?

I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever ...