Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [missing-data]

The tag has no usage guidance.

9
votes
2answers
2k views

What to do when testing data has less features than training data?

Let's say we are predicting the sales of a shop and my training data has two sets of features: One about the store sales with the dates (the field "Store" is not unique) One about the store types (...
6
votes
5answers
2k views

filling missing data with other than mean values

What all options are available for filling missing data ? One obvious choice is the mean, but if the percentage of missing data is large, it will decrease the accuracy. So how do we deal with ...
6
votes
1answer
1k views

Naive Bayes Should generate prediction given missing features (scikit learn)

Seeing that Naive Bayes uses probability to make a prediction, and treats features as being conditionally independent of each other, then it makes sense that the model can still make a prediction ...
5
votes
5answers
72k views

How to replace NA values with another value in factors in R? [closed]

I have a factor variable in my data frame with values where in the original CSV "NA" was intended to mean simply "None", not missing data. Hence I want replace every value in the given column with "...
5
votes
4answers
2k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
5
votes
1answer
306 views

Computationally Inexpensive Imputation Techniques in R

I have a large data-frame (155257 x 21 to be specific) with only a few missing values. Say, some 2.16% of the values need to be imputed. The values are floating point numbers. I'd like to use a ...
4
votes
2answers
649 views

How to handle missing data for machine learning

I'm trying to come up with a data structure to predict water visibility in a lake. I have some measured samples but would like to take other features into the equation. As an example, I would like to ...
4
votes
1answer
129 views

How to fix inconsistent (variable spelling) categorical data and “fill in” missing data

I am a newbie data science engr. My first challenge is to (1) normalize inconsistent values in categorical features and (2) fill any missing information. To describe inconsistency lets say we have a ...
4
votes
2answers
1k views

Handling many missing values

I have a dataset I want to perform multivariate linear regression to it. The dimensions of the dataset are 832085 rows and 11 columns. The data are quite messy and given the size and my lack of ...
3
votes
1answer
249 views

Predicting Missing Features

I have "millions" of items each having N binary features. When a feature is "0" it could be that the information is simply missing. So, given the data with the currently observed 1's, I would like to ...
3
votes
2answers
136 views

How does the naive Bayes classifier handle missing data in testing?

Assume that a classier has been trained already (no missing training data), but a prediction has been requested based on an observation that does not include every feature. How can we handle this ...
3
votes
1answer
79 views

Correlation with missing values. Is least squares an acceptable option?

I have been tasked with finding a correlation matrix for a lot of variables. Many of them have missing values. I read here that pairwise deletion may not be the best way of dealing with this situation,...
3
votes
1answer
136 views

Handling missing values to optimize polynomial features

I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on ...
3
votes
1answer
59 views

Setting “missing” distance values to zero when training a neural network

Not sure if missing values is the right name to use here. I want to train a DNN on data given by a sensor. The sensor gives the (x,y) coordinates of the founded objects. The sensor can keep track of ...
3
votes
1answer
24 views

How to deal with Missing Not at Random Data for k-means clustering?

I am running k-means clustering on a customer dataset. One of the available demographic fields is inferred homevalue, represented as an integer. This field has value 0 when it's inferred that the ...
2
votes
3answers
183 views

Cluster analysis as an associative model?

I have a set of data with many samples and many features, but where half of the data is missing one variable (call it A), which is composed of four categories. Based on the half of data which has A, I ...
2
votes
1answer
163 views

Data prepration for logistic regression : Value either “not available” or a “year”

I have some data of houses that have been renovated. In my data there is one column (among others) that captures this information. It is either "-1" if there has not been yet any renovation, or the ...
2
votes
1answer
432 views

Missing outputs in multiple-output neural net

I am looking at a task, where I want to predict multiple things from an image (an animal's breed [categorical], age [continuous number] and gender [categorical]). Unsurprisingly, my first thought was ...
2
votes
1answer
227 views

Replacing missing value by class conditional mean

I have two classes, $p(x|y=0)$ and $p(x|y=1)$ with ${{\mu }_{0}}$ and ${{\mu }_{1}}$ as mean and shared covariance matrix $\Sigma $. Now, I have a missing feature ${{x}_{n}}$ for a particular ...
2
votes
2answers
2k views

Methodologies for predicting missing data

I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset. For example: I have a dataset which contains information about a person. This can be ...
2
votes
1answer
193 views

How to treat missing data for survival analysis

I have a dataset consisting of questionnaires from patient survey data. There are around 10 questions which are asked during several stages of treatment like during first day of visit, after a week, ...
2
votes
4answers
2k views

Scikit Learn Missing Data - Categorical values

I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier) ...
2
votes
1answer
109 views

How to fill in missing value of the mean of the other columns?

I had a movie dataset including 'budget' and 'genres' attributes. I'd like to fill in the missing value of budget with the mean budget of each genre. I first create two dataframes with or without ...
2
votes
2answers
1k views

How can I handle missing categorical data that has significance?

I have a data set that is highly categorical and has a lot of missing values. For instance: ...
2
votes
1answer
21 views

Training on data with inherently non-applicable data cells

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense. Chemical Dataset Contains thousands of rows of chemical mixtures with many columns of ...
2
votes
2answers
84 views

What is the difference between Missing at Random and Missing not at Random data?

I have been working with a dataset where the missing data seem to following a few particular patterns. I have gone through a lot websites and articles related to missing data but I haven't been able ...
2
votes
1answer
126 views

Toolbox for handling NaNs in Python 2.7

Is there a good toolbox for handling and analyzing missing values in Python 2.7? There is a good toolbox for doing this in Python 3.6 here (missingno): https://github.com/ResidentMario/missingno I ...
2
votes
1answer
2k views

Missing data imputation with KNN

-1 down vote favorite I have a dataset including missing data for most of the variables. Assume the dataset is as follows: ...
2
votes
1answer
176 views

Missing Categorical Features - no imputation

I've been reading about how to approach missing categorical features in test data, and the most common approach is to use imputation - for example using the last known value or getting the majority ...
2
votes
2answers
212 views

Feature scaling data with missing values

am interested in doing some feature scaling to try and tease out something from my data (box plots by outcome show that the 25/50/75 quantiles are very similar; certain variables have more "outliers" ...
2
votes
1answer
29 views

Cluster Analysis - Comparing Same Individuals Clustered Across Different Datasets with different features

I have an interesting problem, and I think my Google is failing me since I can't find the same problem anywhere. I have a set of individuals. I have 4 different datasets, with (some) to (all) of ...
2
votes
2answers
26 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
2
votes
0answers
26 views

Detect Missing Records in Dataset

I have a dataset that contains several measures from various widgets on a daily basis. While the widgets remain relatively stable over time, sometimes there are legitimate reasons for one to disappear ...
2
votes
2answers
20 views

Dealing with diverse groups in regression

What happens if a certain dataset contains different "groups" that follow different linear models? For example, let's imagine that examining the scatterplot of a certain feature $x_i$ against $y$ we ...
1
vote
2answers
49 views

Column With Many Missing Values (36%)

Hello this is my first machine learning project, I got a dataset with 18.000 rows and I have a column with 4244 values missing. I don't know why the values are missing since when it's appropriate ...
1
vote
1answer
795 views

Fill missing values AND normalise

I have two columns of training data for a neural net which are missing values. (There are many other columns which aren't missing values.) For example ...
1
vote
2answers
224 views

How would one impute missing values for a Discrete variable?

How would one imputing missing values (without using the mode) for a discrete variable, e.g. a variable corresponding to a count.
1
vote
1answer
63 views

Data that's not missing is called…?

Is there a standard term for data that's not missing? I.e. is it called non-missing, present, or something else? Thanks!
1
vote
1answer
80 views

Using predictive modelling for temperature data set

I am absolutely new to this area of predictive modelling in data science. I am not able to understand how and what modelling techniques do we use? Does it depend on the data type? Does it depend on ...
1
vote
1answer
50 views

R mice doesn't give a 'valid' sollution

EDITED: See below for additional information.. TL;DR: How can I add missing data in a dataset like the sample in a way that it doesn't deviate much from the original dataset. ORIGINAL: I have a ...
1
vote
1answer
223 views

Imputation missing values other than using Mean, Median in python

I heard that Mean, Median isn't the best way to impute the missing values, why would that be? In my scenario, I have data like this ...
1
vote
1answer
79 views

Does encoding missing data with fixed values help in classification?

I have a lot of missing values for some variables in my data (70-80%). I have seen some people deal with missing values this way: encode the variable with missing values as 0 or 1. Where 0 is the ...
1
vote
0answers
19 views

How to incorporate an attribute that only exists in some observations?

In a binary classification problem, some of my observations have an event that occurs. I can, obviously, add a 1/0 flag if the event occurs ("event_occurred" in the data below). However, my intuition ...
1
vote
0answers
12 views

Finding features for missing classes

I have some data with labels from n-1 classes. There is no datapoint with the n-th class in the data but we know that data from the n-th exist. How can we generate data from the missing n+1-the class? ...
1
vote
1answer
185 views

How to deal with missing data for Bernoulli Naive Bayes?

I am dealing with a dataset of categorical data that looks like this: ...
1
vote
1answer
958 views

Dealing with NaN (missing) values for Logistic Regression- Best practices?

I am working with a data-set of patient information and trying to calculate the Propensity Score from the data using MATLAB. After removing features with many missing values, I am still left with ...
1
vote
1answer
47 views

Missing value in continuous variable: Indicator variable vs. Indicator value

Most data has missing values, and as far as I'm aware, these are the options: Imputation (mean, hot-deck, etc.) Indicator variable. A categorical variable that tells what type the primary variable is....
1
vote
0answers
18 views

Linear Model: How to deal with predictors with a lot of missing/small values?

I have a linear model used for prediction, with around 30 predictors, which are car usage rate as in percentage, across different zip codes. All these predictors have the same unit, as they are all ...
1
vote
1answer
115 views

Missing population values in census data

I have population data from Census.gov: Total US population by age by year from 1940 through 2010 Depending on the range of decades, the data is missing discrete population values for ages greater ...