Questions tagged [missing-data]

Missing data is a problem that arises in data science when some data contained in rows or columns may be missing or unavailable for some samples in a dataset.

Filter by
Sorted by
Tagged with
13
votes
2answers
4k views

What to do when testing data has less features than training data?

Let's say we are predicting the sales of a shop and my training data has two sets of features: One about the store sales with the dates (the field "Store" is not unique) One about the store types (...
11
votes
4answers
31k views

How to use SimpleImputer Class to replace missing values with mean values using Python?

This is my code ...
10
votes
4answers
836 views

How to impute Missing values not the usual way?

I have a dataset of 4712 records working on binary classification. Label 1 is 33% and Label 0 is 67%. I can't drop records because my sample is already small. Because there are few columns which has ...
8
votes
5answers
3k views

Filling missing data with other than mean values

What are all the options available for filling in missing data? One obvious choice is the mean, but if the percentage of missing data is large, it will decrease the accuracy. So how do we deal with ...
7
votes
2answers
3k views

Fill missing values AND normalise

I have two columns of training data for a neural net which are missing values. (There are many other columns which aren't missing values.) For example ...
7
votes
1answer
2k views

Naive Bayes Should generate prediction given missing features (scikit learn)

Seeing that Naive Bayes uses probability to make a prediction, and treats features as being conditionally independent of each other, then it makes sense that the model can still make a prediction ...
6
votes
2answers
319 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
5
votes
4answers
3k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
4
votes
5answers
132k views

How to replace NA values with another value in factors in R? [closed]

I have a factor variable in my data frame with values where in the original CSV "NA" was intended to mean simply "None", not missing data. Hence I want replace every value in the given column with "...
4
votes
3answers
1k views

What predictive model to use to impute Gender?

My data looks like this: birth_date has 634,990 missing values gender has 328,849 missing values Both of these are a ...
4
votes
1answer
2k views

How do GBM algorithms handle missing data?

How do algorithms GBM algorithms, such as XGBoost or LightGBM handle NaN values? I know that they learn how to replace NaN values with other values but my question is: How do they do it exactly?
4
votes
1answer
870 views

Missing outputs in multiple-output neural net

I am looking at a task, where I want to predict multiple things from an image (an animal's breed [categorical], age [continuous number] and gender [categorical]). Unsurprisingly, my first thought was ...
4
votes
4answers
3k views

Scikit Learn Missing Data - Categorical values

I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier) ...
4
votes
1answer
94 views

Why removing rows with NA values from the majority class improves model performance

I have an imbalanced dataset like so: df['y'].value_counts(normalize=True) * 100 ...
4
votes
2answers
2k views

What is the difference between Missing at Random and Missing not at Random data?

I have been working with a dataset where the missing data seem to following a few particular patterns. I have gone through a lot websites and articles related to missing data but I haven't been able ...
4
votes
1answer
422 views

Computationally Inexpensive Imputation Techniques in R

I have a large data-frame (155257 x 21 to be specific) with only a few missing values. Say, some 2.16% of the values need to be imputed. The values are floating point numbers. I'd like to use a ...
4
votes
2answers
2k views

Handling many missing values

I have a dataset I want to perform multivariate linear regression to it. The dimensions of the dataset are 832085 rows and 11 columns. The data are quite messy and given the size and my lack of ...
4
votes
1answer
1k views

How to deal with missing data for Bernoulli Naive Bayes?

I am dealing with a dataset of categorical data that looks like this: ...
3
votes
2answers
782 views

How to handle missing data for machine learning

I'm trying to come up with a data structure to predict water visibility in a lake. I have some measured samples but would like to take other features into the equation. As an example, I would like to ...
3
votes
1answer
571 views

Predicting Missing Features

I have "millions" of items each having N binary features. When a feature is "0" it could be that the information is simply missing. So, given the data with the currently observed 1's, I would like to ...
3
votes
1answer
44 views

How data are prepared during training, testing and in production?

Most of real world datasets have features with missing values. Replacing missing values with an appropriate value such as its mean, is considered as a good step in feature engineering. Some times we ...
3
votes
2answers
576 views

How does the naive Bayes classifier handle missing data in testing?

Assume that a classier has been trained already (no missing training data), but a prediction has been requested based on an observation that does not include every feature. How can we handle this ...
3
votes
1answer
638 views

How to treat missing data for survival analysis

I have a dataset consisting of questionnaires from patient survey data. There are around 10 questions which are asked during several stages of treatment like during first day of visit, after a week, ...
3
votes
1answer
4k views

How to fill in missing value of the mean of the other columns?

I had a movie dataset including 'budget' and 'genres' attributes. I'd like to fill in the missing value of budget with the mean budget of each genre. I first create two dataframes with or without ...
3
votes
1answer
967 views

Handling missing values to optimize polynomial features

I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on ...
3
votes
1answer
6k views

Dealing with NaN (missing) values for Logistic Regression- Best practices?

I am working with a data-set of patient information and trying to calculate the Propensity Score from the data using MATLAB. After removing features with many missing values, I am still left with ...
3
votes
1answer
137 views

Correlation with missing values. Is least squares an acceptable option?

I have been tasked with finding a correlation matrix for a lot of variables. Many of them have missing values. I read here that pairwise deletion may not be the best way of dealing with this situation,...
3
votes
1answer
309 views

Missing Categorical Features - no imputation

I've been reading about how to approach missing categorical features in test data, and the most common approach is to use imputation - for example using the last known value or getting the majority ...
3
votes
1answer
184 views

How to fix inconsistent (variable spelling) categorical data and “fill in” missing data

I am a newbie data science engr. My first challenge is to (1) normalize inconsistent values in categorical features and (2) fill any missing information. To describe inconsistency lets say we have a ...
3
votes
1answer
38 views

Handling rows with 2 lines of data

My dataframe looks like this : there are some rows ( example : 297) where the "Price" column has two values ( Plugs and Quarts) , I have filled the Nans with the previous row since it belongs to the ...
3
votes
1answer
40 views

Cluster Analysis - Comparing Same Individuals Clustered Across Different Datasets with different features

I have an interesting problem, and I think my Google is failing me since I can't find the same problem anywhere. I have a set of individuals. I have 4 different datasets, with (some) to (all) of ...
3
votes
1answer
89 views

Setting “missing” distance values to zero when training a neural network

Not sure if missing values is the right name to use here. I want to train a DNN on data given by a sensor. The sensor gives the (x,y) coordinates of the founded objects. The sensor can keep track of ...
3
votes
0answers
39 views

Detect Missing Records in Dataset

I have a dataset that contains several measures from various widgets on a daily basis. While the widgets remain relatively stable over time, sometimes there are legitimate reasons for one to disappear ...
2
votes
3answers
196 views

Cluster analysis as an associative model?

I have a set of data with many samples and many features, but where half of the data is missing one variable (call it A), which is composed of four categories. Based on the half of data which has A, I ...
2
votes
1answer
169 views

Data prepration for logistic regression : Value either “not available” or a “year”

I have some data of houses that have been renovated. In my data there is one column (among others) that captures this information. It is either "-1" if there has not been yet any renovation, or the ...
2
votes
2answers
3k views

How can I handle missing categorical data that has significance?

I have a data set that is highly categorical and has a lot of missing values. For instance: ...
2
votes
2answers
5k views

Methodologies for predicting missing data

I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset. For example: I have a dataset which contains information about a person. This can be ...
2
votes
1answer
397 views

Replacing missing value by class conditional mean

I have two classes, $p(x|y=0)$ and $p(x|y=1)$ with ${{\mu }_{0}}$ and ${{\mu }_{1}}$ as mean and shared covariance matrix $\Sigma $. Now, I have a missing feature ${{x}_{n}}$ for a particular ...
2
votes
1answer
33 views

Missing at random vs missing not at random: What if it is both? (Does one imply the other?)

My understanding is that: Missing at random: Whether or not a variable's value is missing is dependent on the values of the other variables. Missing not at random: When the propensity for a variable'...
2
votes
1answer
107 views

Making predictions with missing numeric data

I don't know much about how to deal with missing data. When the data is categorical it doesn't seem too bad, if I one-hot encode it without dropping any of the actual categories, the missing data is ...
2
votes
1answer
36 views

Training on data with inherently non-applicable data cells

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense. Chemical Dataset Contains thousands of rows of chemical mixtures with many columns of ...
2
votes
2answers
2k views

Imputation missing values other than using Mean, Median in python

I heard that Mean, Median isn't the best way to impute the missing values, why would that be? In my scenario, I have data like this ...
2
votes
3answers
91 views

Handling categorical missing values ML

I have gone through this regarding handling missing values in categorical data. Dataset has about 6 categorical columns with ...
2
votes
1answer
53 views

Could I add a one hot encoding to each feature representing “has data” versus “has no data”

I have a data set that has some holes in it. I was wondering if I could add two columns for each feature representing this feature has data and ...
2
votes
3answers
86 views

Dealing with no data

I am working on predictive maintenance and get temperature data from assets. In few months or few days asset remains down and we do not get temperature value. In this scenario i cannot fill data with ...
2
votes
2answers
32 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
2
votes
1answer
385 views

Toolbox for handling NaNs in Python 2.7

Is there a good toolbox for handling and analyzing missing values in Python 2.7? There is a good toolbox for doing this in Python 3.6 here (missingno): https://github.com/ResidentMario/missingno I ...
2
votes
1answer
2k views

Missing data imputation with KNN

-1 down vote favorite I have a dataset including missing data for most of the variables. Assume the dataset is as follows: ...
2
votes
2answers
414 views

Feature scaling data with missing values

am interested in doing some feature scaling to try and tease out something from my data (box plots by outcome show that the 25/50/75 quantiles are very similar; certain variables have more "outliers" ...
2
votes
1answer
55 views

How to treat patients without events in time-to-event analysis?

I'm working with longitudinal data for a series of patients. Duration of followup on a patient-level is non-uniform. Patients can either experience a discrete event (e.g., a heart attack) or never ...