Questions tagged [data-imputation]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
1answer
23 views

How to impute missing value in Test Set using a custom Imputer created on training dataset

I am working on a toy project to predict claims. One of the input features has null values on which I have applied a custom imputation technique. Under this technique, I replaced missing values with ...
0
votes
0answers
24 views

Impute missing values in feature column on the basis of Target column

I am working on a toy project for insurance claim prediction. In the input data for one of the feature (numeric data type) half of the values are missing. My target variable is binary which indicates ...
0
votes
0answers
7 views

How to do pixel mean imputation [closed]

I am working on this paper and I want to implement experiment. https://arxiv.org/pdf/2010.13914.pdf In the paper, a 13x13 region is randomly chosen and mean imputation process is performed at pre-...
0
votes
1answer
32 views

K-Fold cross validation and data leakage

I want to do K-Fold cross validation and also I want to do normalization or feature scaling for each fold. So let's say we have k folds. At each step we take one fold as validation set and the ...
0
votes
0answers
10 views

Dropping Missing Observations under MAR Assumption

Some of the outcome data in my data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing and ...
1
vote
0answers
25 views

How to handle large systematic missing data in time series?

I have this time series, where on the weekends, the dependent variable values are missing. It's only a time series, I do not have any exogenous regressors/features. The dependent variable value is an ...
0
votes
1answer
34 views

sklearn KNN imputation

Can I use sklearn's KNN imputer to fit the model to my training set and impute missing values in the test set using the neighbours from training set ? Is it allowed ? Or , Should I only fit and ...
2
votes
2answers
27 views

Missing value Imputation in dataset

I have two separate files for Testing and Training. In the training data, I am dropping rows that contain too many missing values . But , In the test data , I cannot afford to drop the rows so I have ...
0
votes
2answers
108 views

sklearn - How to create a sequential pipeline

Update: The examples in this post were updated I am reposting this question here after not getting a clear answer in a previous SO post I am looking for a help building a data preprocessing pipleline ...
0
votes
0answers
35 views

sklearn SimpleImputer using mean from different column groups?

I'm looking at the SimpleImputer, in particular in here, and I would like to do the imputation on different columns. My data has 3 different sample groups, and I would like to do ...
1
vote
1answer
116 views

Using sklearn knn imputation on a large dataset

I have a large dataset ~ 1 million rows by 400 features and I want to impute the missing values using sklearn KNNImputer. Trying this off the bat I hit memory problems, but I think I can solve this by ...
0
votes
0answers
20 views

If a single element is missing more than 50% of its feature values, should you just remove it?

I have a simple supervised machine learning problem. My training matrix is MxN, where M is the number of records and N is the number of features. I have 600,000 complete patient records and 300,000 ...
1
vote
1answer
35 views

In supervised learning, are more data entries always better?

I am doing a supervised learning problem and have 600,000 rows of data. I divided it into a training and test set and achieved a high accuracy that I was happy with. However, I had thrown away 300,000 ...
0
votes
1answer
37 views

How to handle sparsely coded features in a dataframe

I have a dataset that contains information regarding diabetes patients, like so: ...
2
votes
1answer
111 views

KNN Imputation utilize mean or mode?

In my current project, I am doing KNN imputation with K = 5 and I am using sklearn.impute.KNNImputer. I have a mix of continuous and nominal variables(encoded as 0/1 or ordinal ones that have been ...
1
vote
1answer
26 views

How can we use mean imputation without violating feature correlation?

Mean imputation is generally bad practice because it doesn’t take into account feature correlation. Imagine we have a table showing age and fitness score and imagine that an eighty-year-old has a ...
1
vote
1answer
51 views

Dealing with missing data

I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both ...
0
votes
1answer
344 views

How to impute using simple imputer (custom function)

I am imputing my data using simple imputer from sklearn. i want to test many different ways of applying transformations to the data. i.e for logisitcic regression i would like to remove nans and ...
2
votes
3answers
78 views

Handling categorical missing values ML

I have gone through this regarding handling missing values in categorical data. Dataset has about 6 categorical columns with ...
1
vote
2answers
218 views

What is the point of using MissingIndicator in Scikit-learn?

I have recently discovered Sklearn's MissingIndicator but still wondering how could it improve the usual machine learning work. Clear that ...
1
vote
0answers
17 views

Fit model function out defined data range

I have asked this on SO but it has not been well accepted because it seems to be more about data science than programming. Let's say I have a set of data ...
0
votes
2answers
45 views

Right order for Data preparation in Machine Learning

For the below mentioned steps of data preparation Outlier detection/treatment Data imputation Data scaling/standardisation Class balancing There are two sub questions Should each of these steps ...
10
votes
5answers
2k views

Please review my sketch of the Machine Learning process

It's amazingly difficult to find an outline of the end-to-end machine learning process. As a total beginner, this lack of information is frustrating, so I decided to try scraping together my own ...
0
votes
0answers
18 views

How to impute missing values for hour of day?

I have a dataset containing a certain amount of bookings, no shows and cancellations. We assume that a cancellation less than 2 days is the same as a no show (since you did not have the time to ...
0
votes
0answers
34 views

Input missing data with interpolation [duplicate]

When dealing with missing values there are several strategies that can be used such as: imputing with the mean, median, a fixed value... I have read that you can do things such as interpolation. ...
0
votes
1answer
82 views

How to impute right-censored data

I have a dataset of vectors representing movement with various characteristics. Some vectors represents the movement that was stopped by external factor and therefore, observed value for length of ...
-1
votes
1answer
116 views

How to handle time series missing values

I have a database of thermal consumption of 100 buildings. Each file has two columns, one is timestamp and the other is usage. My task is to build a prediction model for forecasting the usage for the ...
2
votes
2answers
307 views

What do I do when my column has 50% data missing?

I have a data set i am pre-processing. However in my categorical columns (3 of them) i have "??" in it's place. They constitute 50% of the data. In fact 3 columns have this. My question is how should ...
1
vote
1answer
20 views

Dealing with issues in “test” predictons for single “items” (null values, standardization in place, etc)

I know this is kind of a broad question but I have tried to scour both this forum and the internet in general to no avail for this particular situation. So imagine I have a model trained for which, ...
0
votes
0answers
41 views

Memory error while trying to apply Multivariate imputation by chained equations?

I have a training set consisting of movies data, which includes features like runtime, keywords, cast etc. The keywords columns consist of json collections of keywords used in the movie with ...
0
votes
0answers
54 views

What is the difference between “fit_transform” and “transform” methods when using “SimpleImputer”? [duplicate]

I have following code, I am not able to understand the difference between use of fit_transform() and transform() method in this ...
5
votes
2answers
640 views

Retrieve dropped column names from `sklearn.impute.SimpleImputer`

The SimpleImputer class takes pandas dataframes and returns unlabeled numpy arrays. Which means that the SimpleImputer drops ...
2
votes
1answer
66 views

How to implement single Imputation from conditional distribution?

In [*] page 264, a method of drawing a missing value from a conditional distribution $P(\bf{x}_{mis}|\bf{x}_{obs};\theta)$ which is defined as: I did not find any code implementation of this ...
0
votes
0answers
104 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
0
votes
2answers
5k views

How can I replace outliers with maximum non-outlier value?

I am doing univariate outlier detection in python. When I detect outliers for a variable, I know that the value should be whatever the highest non-outlier value is (i.e., the max if there were no ...
6
votes
2answers
256 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
8
votes
2answers
3k views

Which comes first? Multiple Imputation, Splitting into train/test, or Standardization/Normalization

I am working on a multi-class classification problem, with ~65 features and ~150K instances. 30% of features are categorical and the rest are numerical (continuous). I understand that standardization ...
3
votes
1answer
1k views

Advice on imputing temperature data with StatsModels MICE

This may be a dumb question but I can't figure out how to actually get the values imputed using StatsModels MICE back into my data. I have a dataframe (dfLocal) with hourly temperature records for ...
1
vote
3answers
778 views

Is it right to impute Train and Test set?

I am experimenting with a dataset and I have a couple of columns with high cardinality. So, I performed mean target encoding (given that my dataset had more than 50000 observations). But, before doing ...
4
votes
3answers
933 views

What predictive model to use to impute Gender?

My data looks like this: birth_date has 634,990 missing values gender has 328,849 missing values Both of these are a ...
2
votes
3answers
101 views

Machine Learning, Imputing values that should be blank

Sometimes data sets contain variables that indicate the presence of an event and the value that represented the event. As an example say a teacher wants to predict the grades of his students. Some of ...
3
votes
1answer
631 views

Target Encoding: missing value imputation before or after encoding

I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values. Let's say I have a few continuous features,...
2
votes
1answer
36 views

Training on data with inherently non-applicable data cells

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense. Chemical Dataset Contains thousands of rows of chemical mixtures with many columns of ...
0
votes
1answer
50 views

The proper way to codify Na in a list in R

I am trying to impute missing timeseries present in different dimentions, row by row, on the whole date set. I showed the type of return of na.kalman() and it ...
2
votes
2answers
32 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
2
votes
1answer
2k views

How to handle missing data data in dependent variable?

I'm solving a ML problem statement where there are around 40k records in the dataset. A dependent variable is given in the question (There are many independent variables). But there are some 2k ...
2
votes
2answers
361 views

How to fill missing numeric if any value in a subset is missing, all other columns with the same subset are missing

There is a clear pattern that show for two separate subsets (set of columns); If one value is missing in a column, values of other columns in the same subset are missing for any row. Here is a ...
3
votes
1answer
933 views

Handling missing values to optimize polynomial features

I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on ...
0
votes
1answer
45 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...
0
votes
1answer
653 views

pandas: How to impute the categorical column by the nearest neighbors?

I've a categorical column with values such as right('r'), left('l') and straight('s'). I expect these to have a continuum periods in the data and want to impute nans with the most plausible value in ...