Questions tagged [missing-data]

Missing data is a problem that arises in data science when some data contained in rows or columns may be missing or unavailable for some samples in a dataset.

Filter by
Sorted by
Tagged with
0
votes
1answer
8 views

Error in lmest: missing data in the covariates affecting the initial probabilities are not allowed

I'm running an LM model using the LMest package available in R. The dataset contains NO missing values. pct_miss(df_long) [1] 0 n_miss(df_long) [1] 0 The lmest function with no covariates works fine....
0
votes
1answer
28 views

Can I deal with a missing not at random column by creating a new column? (Feature engineering)

Task: Binary classification Example problem: Let's say we have two feature columns A and B. A has no nulls and is a binary column if a user completed an action (=1), 0 if they didn't. For all users ...
0
votes
1answer
25 views

the missForest function in R doesn't work

I'm trying to use the function missForest() of the library 'missForest' but I always get the same error message. This is the code: libraries: ...
2
votes
1answer
28 views

Do i need to handle missing values before EDA?

I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to ...
1
vote
1answer
158 views

Encode missing data and unseen data

Let's assume that I have a classification problem and all my features are categorical data. I have missing data (and I do not want to do any imputation). Also, I know that I will have some unseen ...
2
votes
1answer
55 views

How to treat patients without events in time-to-event analysis?

I'm working with longitudinal data for a series of patients. Duration of followup on a patient-level is non-uniform. Patients can either experience a discrete event (e.g., a heart attack) or never ...
0
votes
0answers
10 views

How are missing values treated in XGB RF Classifier?

I was exploring Random Forest Classifier in XGBoost listed here : https://xgboost.readthedocs.io/en/latest/python/python_api.html I was wondering how the missing values will be handled in this ...
1
vote
2answers
77 views

Handling NA Values in the Chicago Crime Rate data set

I am doing a little project on the Chicago Crime Rate data set and I noticed that there are over 600,000 NA values, primarily in the location fields. I feel that ...
1
vote
1answer
34 views

Replace Missing Values with Most Frequent number under Condition

I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster ...
2
votes
1answer
47 views

How to deal with Missing Not at Random Data for k-means clustering?

I am running k-means clustering on a customer dataset. One of the available demographic fields is inferred homevalue, represented as an integer. This field has value 0 when it's inferred that the ...
0
votes
0answers
8 views

Do I remove the missing values before performing univariate and bivariate analysis? Is there a general rule?

I have read the answer to this question, but it doesn't quite answer my question whether there is a general rule for dealing with this situation. When performing EDA I find missing values in the ...
4
votes
1answer
94 views

Why removing rows with NA values from the majority class improves model performance

I have an imbalanced dataset like so: df['y'].value_counts(normalize=True) * 100 ...
1
vote
0answers
19 views

Handling missing data - secondary driver characteristics in insurance data

I have an insurance dataset which includes an indicator that indicates whether the policy insures a secondary driver, and the secondary driver's age/sex. Problem is most policies do not have secondary ...
1
vote
1answer
34 views

How to deal with missing values in the survey data to perform Paired sample t test

I have a dataset where I have 100 respondents. Each respondent has to give response on service quality of Health care equipment. Is it providing efficient services to the patients? We have two ...
4
votes
1answer
1k views
2
votes
1answer
224 views

Missing population values in census data

I have population data from Census.gov: Total US population by age by year from 1940 through 2010 Depending on the range of decades, the data is missing discrete population values for ages greater ...
4
votes
4answers
3k views

Scikit Learn Missing Data - Categorical values

I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier) ...
0
votes
1answer
2k views

To calculate unaffected part of the data set with missing values and positive skewness

A dataset has some missing values with positive skewness = 1. It is known that it is spread over 1.5 standard deviation from the median. How much % of data will remain unaffected?
0
votes
1answer
54 views

Dealing with missing data in SVD

I am a newbie to machine learning and I am trying to apply the SVD on the movielens dataset for movie recommendation. I have a movie-user matrix where the row is the user id, the column is the movie ...
0
votes
1answer
259 views

Fill the missing values (NA) in various columns (independently of each other) using imputeTS package (in particular, na_kalman function)

A friend of mine has recently started working on R-studio and is interested in filling the NA values in different columns using the above-mentioned function. Also, since he intends to run a time ...
2
votes
2answers
44 views

Dealing with diverse groups in regression

What happens if a certain dataset contains different "groups" that follow different linear models? For example, let's imagine that examining the scatterplot of a certain feature $x_i$ against $y$ we ...
3
votes
1answer
44 views

How data are prepared during training, testing and in production?

Most of real world datasets have features with missing values. Replacing missing values with an appropriate value such as its mean, is considered as a good step in feature engineering. Some times we ...
2
votes
3answers
135 views

What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation. I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) ...
2
votes
1answer
69 views

How to implement single Imputation from conditional distribution?

In [*] page 264, a method of drawing a missing value from a conditional distribution $P(\bf{x}_{mis}|\bf{x}_{obs};\theta)$ which is defined as: I did not find any code implementation of this ...
1
vote
1answer
22 views

Dealing with missing data in several features at once

Good day, What are the approaches for handling missing data in several features (categorical and continuous) at once? I look through each feature and plotted several histograms of the distribution of ...
0
votes
0answers
10 views

Dropping Missing Observations under MAR Assumption

Some of the outcome data in my data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing and ...
0
votes
1answer
41 views

sklearn KNN imputation

Can I use sklearn's KNN imputer to fit the model to my training set and impute missing values in the test set using the neighbours from training set ? Is it allowed ? Or , Should I only fit and ...
1
vote
2answers
59 views

How to build a model on a dataset having 40% missing values in most of the variables?

I have a huge dataset of 10 million observations but most of the variables are missing for 40% records. There are couple of variables available for the whole dataset such as sic code(Industry category)...
1
vote
1answer
31 views

How to handle a valuable feature that is missing on 99\% of the samples in the data set?

Suppose we have an input feature that is highly predictive of the outcome we want to predict. However, the feature is missing on 99% of the samples in the data set. What is the best way to use this ...
0
votes
0answers
12 views

Positive records : no behavioral data

0 I'm working on a classification model aimed at identifying if behavioral activity within an account (b2b - one account, many contacts) can predict or not an opportunity generation ( a salesperson ...
1
vote
1answer
610 views

How to fill missing values by looking at another row with same value in one column(or more)?

Let's say we have a 6*4 data frame in which third and fourth column contain missing value ...
1
vote
0answers
29 views

How to handle large systematic missing data in time series?

I have this time series, where on the weekends, the dependent variable values are missing. It's only a time series, I do not have any exogenous regressors/features. The dependent variable value is an ...
1
vote
1answer
25 views

Dropping missing rows in two dataframes

I have two files : Test_data - contains the features of a dataset to find predictions for Submission_data - contains two columns : The index column for test data and another column for its ...
1
vote
1answer
41 views

Why we can't Remove features with missing values in Data Preprocessing [closed]

In a Real Time Dataset, There are many missing values available in the Dataset and also we need to deal with data preprocessing. And there are many ways to minimize the problem of missing values ...
0
votes
0answers
21 views

How to deal with a modular set of features? Input data from intermittent sensors

I want to set up a ML problem with input data from sensors. The issue is that the sensors are not active all the time, and the standard methods to deal with missing values (delete row, impute, predict…...
5
votes
4answers
3k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
10
votes
4answers
843 views

How to impute Missing values not the usual way?

I have a dataset of 4712 records working on binary classification. Label 1 is 33% and Label 0 is 67%. I can't drop records because my sample is already small. Because there are few columns which has ...
2
votes
3answers
93 views

Handling categorical missing values ML

I have gone through this regarding handling missing values in categorical data. Dataset has about 6 categorical columns with ...
0
votes
1answer
37 views

How to handle sparsely coded features in a dataframe

I have a dataset that contains information regarding diabetes patients, like so: ...
1
vote
1answer
51 views

Dealing with missing data

I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both ...
4
votes
1answer
2k views

How do GBM algorithms handle missing data?

How do algorithms GBM algorithms, such as XGBoost or LightGBM handle NaN values? I know that they learn how to replace NaN values with other values but my question is: How do they do it exactly?
2
votes
1answer
33 views

Missing at random vs missing not at random: What if it is both? (Does one imply the other?)

My understanding is that: Missing at random: Whether or not a variable's value is missing is dependent on the values of the other variables. Missing not at random: When the propensity for a variable'...
2
votes
2answers
2k views

Imputation missing values other than using Mean, Median in python

I heard that Mean, Median isn't the best way to impute the missing values, why would that be? In my scenario, I have data like this ...
1
vote
1answer
28 views

How can we use mean imputation without violating feature correlation?

Mean imputation is generally bad practice because it doesn’t take into account feature correlation. Imagine we have a table showing age and fitness score and imagine that an eighty-year-old has a ...
1
vote
2answers
294 views

How to decide on using xgboost with imputation or without it and keeping missing values?

I have a large genetic dataset that I am using xgboost on to score most likely disease causing genes - giving the genes a score between 0-1 of likelihood. I try to avoid features with a lot of ...
1
vote
0answers
14 views

Non-monotone missing data, and inverse probability weighting

I'm having difficulty identifying whether or not my missing data pattern is 'monotone'. I have two variables with missing data, and the missing data patterns in each variable do not completely ...
0
votes
1answer
18 views

How to deal with missing continuous data

So I have a dataset where I have a continuous variable for only about 10% of the entries. How would you incorporate this in a model. Imputing does not make much sense to me, because there are so few ...
3
votes
1answer
38 views

Handling rows with 2 lines of data

My dataframe looks like this : there are some rows ( example : 297) where the "Price" column has two values ( Plugs and Quarts) , I have filled the Nans with the previous row since it belongs to the ...
0
votes
0answers
10 views

Which Coefficients Should Be Used For Imputation of Validation Data

Suppose missing values in the training set imputed by using a regression model. In testing phase, should coefficients from regression model which used for imputation of training set be used to impute ...