Questions tagged [missing-data]

Missing data is a problem that arises in data science when some data contained in rows or columns may be missing or unavailable for some samples in a dataset.

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Error in lmest: missing data in the covariates affecting the initial probabilities are not allowed

I'm running an LM model using the LMest package available in R. The dataset contains NO missing values. pct_miss(df_long) [1] 0 n_miss(df_long) [1] 0 The lmest function with no covariates works fine....
2
votes
1answer
55 views

How to treat patients without events in time-to-event analysis?

I'm working with longitudinal data for a series of patients. Duration of followup on a patient-level is non-uniform. Patients can either experience a discrete event (e.g., a heart attack) or never ...
0
votes
0answers
10 views

How are missing values treated in XGB RF Classifier?

I was exploring Random Forest Classifier in XGBoost listed here : https://xgboost.readthedocs.io/en/latest/python/python_api.html I was wondering how the missing values will be handled in this ...
0
votes
1answer
28 views

Can I deal with a missing not at random column by creating a new column? (Feature engineering)

Task: Binary classification Example problem: Let's say we have two feature columns A and B. A has no nulls and is a binary column if a user completed an action (=1), 0 if they didn't. For all users ...
0
votes
0answers
8 views

Do I remove the missing values before performing univariate and bivariate analysis? Is there a general rule?

I have read the answer to this question, but it doesn't quite answer my question whether there is a general rule for dealing with this situation. When performing EDA I find missing values in the ...
0
votes
1answer
24 views

the missForest function in R doesn't work

I'm trying to use the function missForest() of the library 'missForest' but I always get the same error message. This is the code: libraries: ...
2
votes
1answer
28 views

Do i need to handle missing values before EDA?

I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to ...
1
vote
0answers
19 views

Handling missing data - secondary driver characteristics in insurance data

I have an insurance dataset which includes an indicator that indicates whether the policy insures a secondary driver, and the secondary driver's age/sex. Problem is most policies do not have secondary ...
4
votes
1answer
94 views

Why removing rows with NA values from the majority class improves model performance

I have an imbalanced dataset like so: df['y'].value_counts(normalize=True) * 100 ...
1
vote
1answer
34 views

How to deal with missing values in the survey data to perform Paired sample t test

I have a dataset where I have 100 respondents. Each respondent has to give response on service quality of Health care equipment. Is it providing efficient services to the patients? We have two ...
1
vote
1answer
34 views

Replace Missing Values with Most Frequent number under Condition

I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster ...
3
votes
1answer
44 views

How data are prepared during training, testing and in production?

Most of real world datasets have features with missing values. Replacing missing values with an appropriate value such as its mean, is considered as a good step in feature engineering. Some times we ...
1
vote
1answer
22 views

Dealing with missing data in several features at once

Good day, What are the approaches for handling missing data in several features (categorical and continuous) at once? I look through each feature and plotted several histograms of the distribution of ...
0
votes
0answers
10 views

Dropping Missing Observations under MAR Assumption

Some of the outcome data in my data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing and ...
1
vote
2answers
59 views

How to build a model on a dataset having 40% missing values in most of the variables?

I have a huge dataset of 10 million observations but most of the variables are missing for 40% records. There are couple of variables available for the whole dataset such as sic code(Industry category)...
1
vote
1answer
30 views

How to handle a valuable feature that is missing on 99\% of the samples in the data set?

Suppose we have an input feature that is highly predictive of the outcome we want to predict. However, the feature is missing on 99% of the samples in the data set. What is the best way to use this ...
0
votes
0answers
12 views

Positive records : no behavioral data

0 I'm working on a classification model aimed at identifying if behavioral activity within an account (b2b - one account, many contacts) can predict or not an opportunity generation ( a salesperson ...
1
vote
0answers
29 views

How to handle large systematic missing data in time series?

I have this time series, where on the weekends, the dependent variable values are missing. It's only a time series, I do not have any exogenous regressors/features. The dependent variable value is an ...
0
votes
1answer
40 views

sklearn KNN imputation

Can I use sklearn's KNN imputer to fit the model to my training set and impute missing values in the test set using the neighbours from training set ? Is it allowed ? Or , Should I only fit and ...
1
vote
1answer
25 views

Dropping missing rows in two dataframes

I have two files : Test_data - contains the features of a dataset to find predictions for Submission_data - contains two columns : The index column for test data and another column for its ...
1
vote
1answer
41 views

Why we can't Remove features with missing values in Data Preprocessing [closed]

In a Real Time Dataset, There are many missing values available in the Dataset and also we need to deal with data preprocessing. And there are many ways to minimize the problem of missing values ...
0
votes
0answers
21 views

How to deal with a modular set of features? Input data from intermittent sensors

I want to set up a ML problem with input data from sensors. The issue is that the sensors are not active all the time, and the standard methods to deal with missing values (delete row, impute, predict…...
0
votes
1answer
37 views

How to handle sparsely coded features in a dataframe

I have a dataset that contains information regarding diabetes patients, like so: ...
2
votes
1answer
33 views

Missing at random vs missing not at random: What if it is both? (Does one imply the other?)

My understanding is that: Missing at random: Whether or not a variable's value is missing is dependent on the values of the other variables. Missing not at random: When the propensity for a variable'...
1
vote
1answer
28 views

How can we use mean imputation without violating feature correlation?

Mean imputation is generally bad practice because it doesn’t take into account feature correlation. Imagine we have a table showing age and fitness score and imagine that an eighty-year-old has a ...
1
vote
1answer
51 views

Dealing with missing data

I have a question about data cleaning. I am a novice and have just started learning in this field so please pardon my ignorance. Suppose there are two columns and based on some samples taken from both ...
1
vote
2answers
292 views

How to decide on using xgboost with imputation or without it and keeping missing values?

I have a large genetic dataset that I am using xgboost on to score most likely disease causing genes - giving the genes a score between 0-1 of likelihood. I try to avoid features with a lot of ...
1
vote
0answers
14 views

Non-monotone missing data, and inverse probability weighting

I'm having difficulty identifying whether or not my missing data pattern is 'monotone'. I have two variables with missing data, and the missing data patterns in each variable do not completely ...
0
votes
1answer
18 views

How to deal with missing continuous data

So I have a dataset where I have a continuous variable for only about 10% of the entries. How would you incorporate this in a model. Imputing does not make much sense to me, because there are so few ...
0
votes
0answers
10 views

Which Coefficients Should Be Used For Imputation of Validation Data

Suppose missing values in the training set imputed by using a regression model. In testing phase, should coefficients from regression model which used for imputation of training set be used to impute ...
2
votes
3answers
91 views

Handling categorical missing values ML

I have gone through this regarding handling missing values in categorical data. Dataset has about 6 categorical columns with ...
1
vote
1answer
35 views

How to handle columns with large/infinte values in dataset for ML classification

Computed a column using a formula (formula does't involve any log functions, just a group by with .sum()), but as expected this column would result in ...
1
vote
1answer
588 views

How to fill missing values by looking at another row with same value in one column(or more)?

Let's say we have a 6*4 data frame in which third and fourth column contain missing value ...
0
votes
2answers
575 views

Can Random Forest regressor or Decision trees handle missing values and outliers? [closed]

I have below assumptions about RF & Decision trees in general, please correct me if the assumptions are incorrect. It takes care of missing values It handles outliers It handles skewness in the ...
3
votes
1answer
38 views

Handling rows with 2 lines of data

My dataframe looks like this : there are some rows ( example : 297) where the "Price" column has two values ( Plugs and Quarts) , I have filled the Nans with the previous row since it belongs to the ...
0
votes
0answers
10 views

The data set missing values and percentage of data that would remain unaffected

Can not understand question and answer to it given on DS questions site (see link below): Q3. You are given a data set. The data set has missing values which spread along 1 standard deviation from ...
1
vote
2answers
57 views

I intend to do classification modelling, but my target variable has only one value

Currently I have a dataset and I am trying to predict whether someone will default on their bank loan. The dataset is quite tricky. It covers those who have defaulted in the past, but is also ...
0
votes
1answer
54 views

Dealing with missing data in SVD

I am a newbie to machine learning and I am trying to apply the SVD on the movielens dataset for movie recommendation. I have a movie-user matrix where the row is the user id, the column is the movie ...
0
votes
0answers
30 views

Handling missing timestamps in LSTM model

I have 10 minute SCADA data of wind turbines and many timestamps are missing in between because of regular shutdown and weather conditions. My objective is to predict gearbox failure. How can I handle ...
0
votes
1answer
37 views

Should we first remove the unwanted independent variable or split the data into test train data?

I wanted to know if we are dropping any independent variable as it has too many missing values(~75% or more) from the data then should we do it before splitting the data or after splitting the data
-1
votes
1answer
173 views

How to handle time series missing values

I have a database of thermal consumption of 100 buildings. Each file has two columns, one is timestamp and the other is usage. My task is to build a prediction model for forecasting the usage for the ...
1
vote
1answer
21 views

Dealing with issues in “test” predictons for single “items” (null values, standardization in place, etc)

I know this is kind of a broad question but I have tried to scour both this forum and the internet in general to no avail for this particular situation. So imagine I have a model trained for which, ...
1
vote
0answers
23 views

How to handle and report patient characteristic statistic with missing data in essay?

I'm now working on an clinical trial essay with some doctors. When generating patient characteristic statistic form, I found basic patient characteristic data has some missing. The missing datas are ...
0
votes
0answers
55 views

What is the difference between “fit_transform” and “transform” methods when using “SimpleImputer”? [duplicate]

I have following code, I am not able to understand the difference between use of fit_transform() and transform() method in this ...
10
votes
4answers
836 views

How to impute Missing values not the usual way?

I have a dataset of 4712 records working on binary classification. Label 1 is 33% and Label 0 is 67%. I can't drop records because my sample is already small. Because there are few columns which has ...
1
vote
1answer
23 views

Replacing missing values with mean of feature calculated from previously replaced values

I am not sure how to ask this, but I will try my best. I have replaced some missing values in a feature with the mean of the feature conditional on a second categorical feature. However, not all ...
4
votes
1answer
2k views

How do GBM algorithms handle missing data?

How do algorithms GBM algorithms, such as XGBoost or LightGBM handle NaN values? I know that they learn how to replace NaN values with other values but my question is: How do they do it exactly?
2
votes
1answer
107 views

Making predictions with missing numeric data

I don't know much about how to deal with missing data. When the data is categorical it doesn't seem too bad, if I one-hot encode it without dropping any of the actual categories, the missing data is ...
2
votes
0answers
22 views

Distinction of different types of missing values is lost after importing data from SPSS into R

I've got a file containing survey data in SPSS. There are 3 types of missing values defined: invalid (coded as 900), not applicable (990), not filled in (999). After importing the SPSS file into R ...