Questions tagged [missing-data]

Missing data is a problem that arises in data science when some data contained in rows or columns may be missing or unavailable for some samples in a dataset.

41 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
1answer
1k views

How to deal with missing data for Bernoulli Naive Bayes?

I am dealing with a dataset of categorical data that looks like this: ...
3
votes
0answers
39 views

Detect Missing Records in Dataset

I have a dataset that contains several measures from various widgets on a daily basis. While the widgets remain relatively stable over time, sometimes there are legitimate reasons for one to disappear ...
2
votes
1answer
55 views

How to treat patients without events in time-to-event analysis?

I'm working with longitudinal data for a series of patients. Duration of followup on a patient-level is non-uniform. Patients can either experience a discrete event (e.g., a heart attack) or never ...
2
votes
1answer
27 views

Do i need to handle missing values before EDA?

I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to ...
2
votes
0answers
22 views

Distinction of different types of missing values is lost after importing data from SPSS into R

I've got a file containing survey data in SPSS. There are 3 types of missing values defined: invalid (coded as 900), not applicable (990), not filled in (999). After importing the SPSS file into R ...
2
votes
1answer
68 views

How to implement single Imputation from conditional distribution?

In [*] page 264, a method of drawing a missing value from a conditional distribution $P(\bf{x}_{mis}|\bf{x}_{obs};\theta)$ which is defined as: I did not find any code implementation of this ...
2
votes
2answers
44 views

Dealing with diverse groups in regression

What happens if a certain dataset contains different "groups" that follow different linear models? For example, let's imagine that examining the scatterplot of a certain feature $x_i$ against $y$ we ...
2
votes
1answer
47 views

How to deal with Missing Not at Random Data for k-means clustering?

I am running k-means clustering on a customer dataset. One of the available demographic fields is inferred homevalue, represented as an integer. This field has value 0 when it's inferred that the ...
2
votes
1answer
224 views

Missing population values in census data

I have population data from Census.gov: Total US population by age by year from 1940 through 2010 Depending on the range of decades, the data is missing discrete population values for ages greater ...
2
votes
0answers
1k views

predict() returns NA values

I have the problem. predict() method returns NA. My plan is: Read data from file and separate data to 2 sets: test and train Remove column with NA fraction over 95% Replace NA values with mean value ...
2
votes
3answers
134 views

What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation. I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) ...
1
vote
0answers
19 views

Handling missing data - secondary driver characteristics in insurance data

I have an insurance dataset which includes an indicator that indicates whether the policy insures a secondary driver, and the secondary driver's age/sex. Problem is most policies do not have secondary ...
1
vote
1answer
33 views

Replace Missing Values with Most Frequent number under Condition

I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster ...
1
vote
0answers
28 views

How to handle large systematic missing data in time series?

I have this time series, where on the weekends, the dependent variable values are missing. It's only a time series, I do not have any exogenous regressors/features. The dependent variable value is an ...
1
vote
0answers
14 views

Non-monotone missing data, and inverse probability weighting

I'm having difficulty identifying whether or not my missing data pattern is 'monotone'. I have two variables with missing data, and the missing data patterns in each variable do not completely ...
1
vote
0answers
23 views

How to handle and report patient characteristic statistic with missing data in essay?

I'm now working on an clinical trial essay with some doctors. When generating patient characteristic statistic form, I found basic patient characteristic data has some missing. The missing datas are ...
1
vote
1answer
157 views

Encode missing data and unseen data

Let's assume that I have a classification problem and all my features are categorical data. I have missing data (and I do not want to do any imputation). Also, I know that I will have some unseen ...
1
vote
0answers
20 views

How to incorporate an attribute that only exists in some observations?

In a binary classification problem, some of my observations have an event that occurs. I can, obviously, add a 1/0 flag if the event occurs ("event_occurred" in the data below). However, my intuition ...
1
vote
2answers
77 views

Handling NA Values in the Chicago Crime Rate data set

I am doing a little project on the Chicago Crime Rate data set and I noticed that there are over 600,000 NA values, primarily in the location fields. I feel that ...
1
vote
0answers
22 views

Linear Model: How to deal with predictors with a lot of missing/small values?

I have a linear model used for prediction, with around 30 predictors, which are car usage rate as in percentage, across different zip codes. All these predictors have the same unit, as they are all ...
1
vote
0answers
45 views

Fix missing data by adding another feature instead of using the mean?

I am trying to build a model which predicts whether a user will unsubscribe from a service. There is a particular column which tells the number of hours until a report was written for the user. These ...
1
vote
1answer
459 views

XGBOOST missing_value feature degrades my performance?

I'm training an xgboost model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-...
0
votes
0answers
10 views

How are missing values treated in XGB RF Classifier?

I was exploring Random Forest Classifier in XGBoost listed here : https://xgboost.readthedocs.io/en/latest/python/python_api.html I was wondering how the missing values will be handled in this ...
0
votes
1answer
26 views

Can I deal with a missing not at random column by creating a new column? (Feature engineering)

Task: Binary classification Example problem: Let's say we have two feature columns A and B. A has no nulls and is a binary column if a user completed an action (=1), 0 if they didn't. For all users ...
0
votes
0answers
8 views

Do I remove the missing values before performing univariate and bivariate analysis? Is there a general rule?

I have read the answer to this question, but it doesn't quite answer my question whether there is a general rule for dealing with this situation. When performing EDA I find missing values in the ...
0
votes
0answers
22 views

the missForest function in R doesn't work

I'm trying to use the function missForest() of the library 'missForest' but I always get the same error message. This is the code: libraries: ...
0
votes
0answers
10 views

Dropping Missing Observations under MAR Assumption

Some of the outcome data in my data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing and ...
0
votes
0answers
12 views

Positive records : no behavioral data

0 I'm working on a classification model aimed at identifying if behavioral activity within an account (b2b - one account, many contacts) can predict or not an opportunity generation ( a salesperson ...
0
votes
0answers
21 views

How to deal with a modular set of features? Input data from intermittent sensors

I want to set up a ML problem with input data from sensors. The issue is that the sensors are not active all the time, and the standard methods to deal with missing values (delete row, impute, predict…...
0
votes
0answers
10 views

Which Coefficients Should Be Used For Imputation of Validation Data

Suppose missing values in the training set imputed by using a regression model. In testing phase, should coefficients from regression model which used for imputation of training set be used to impute ...
0
votes
0answers
10 views

The data set missing values and percentage of data that would remain unaffected

Can not understand question and answer to it given on DS questions site (see link below): Q3. You are given a data set. The data set has missing values which spread along 1 standard deviation from ...
0
votes
1answer
54 views

Dealing with missing data in SVD

I am a newbie to machine learning and I am trying to apply the SVD on the movielens dataset for movie recommendation. I have a movie-user matrix where the row is the user id, the column is the movie ...
0
votes
0answers
29 views

Handling missing timestamps in LSTM model

I have 10 minute SCADA data of wind turbines and many timestamps are missing in between because of regular shutdown and weather conditions. My objective is to predict gearbox failure. How can I handle ...
0
votes
1answer
245 views

Fill the missing values (NA) in various columns (independently of each other) using imputeTS package (in particular, na_kalman function)

A friend of mine has recently started working on R-studio and is interested in filling the NA values in different columns using the above-mentioned function. Also, since he intends to run a time ...
0
votes
0answers
28 views

How to handle missing data in a logistic regression?

I am building a model to solve a binary classification task. So far, the input is low dimensional (10 dimensions at most). I need to face the occurrences of missing input. It is my first time at ...
0
votes
0answers
113 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
0
votes
0answers
55 views

Linear/Logistic Regression for unknown values or how to get a good prior for new coefficients

Suppose, we model the probability of making holidays by country and town. The input data are people and how many people actually made holiday in that particular town: ...
0
votes
1answer
2k views

To calculate unaffected part of the data set with missing values and positive skewness

A dataset has some missing values with positive skewness = 1. It is known that it is spread over 1.5 standard deviation from the median. How much % of data will remain unaffected?
0
votes
1answer
45 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...
-1
votes
1answer
166 views

How to handle time series missing values

I have a database of thermal consumption of 100 buildings. Each file has two columns, one is timestamp and the other is usage. My task is to build a prediction model for forecasting the usage for the ...