Questions tagged [missing-data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
9
votes
2answers
2k views

What to do when testing data has less features than training data?

Let's say we are predicting the sales of a shop and my training data has two sets of features: One about the store sales with the dates (the field "Store" is not unique) One about the store types (...
7
votes
1answer
1k views

Naive Bayes Should generate prediction given missing features (scikit learn)

Seeing that Naive Bayes uses probability to make a prediction, and treats features as being conditionally independent of each other, then it makes sense that the model can still make a prediction ...
6
votes
5answers
2k views

Filling missing data with other than mean values

What are all the options available for filling in missing data? One obvious choice is the mean, but if the percentage of missing data is large, it will decrease the accuracy. So how do we deal with ...
5
votes
4answers
2k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
4
votes
5answers
83k views

How to replace NA values with another value in factors in R? [closed]

I have a factor variable in my data frame with values where in the original CSV "NA" was intended to mean simply "None", not missing data. Hence I want replace every value in the given column with "...
4
votes
3answers
153 views

What predictive model to use to impute Gender?

My data looks like this: birth_date has 634,990 missing values gender has 328,849 missing values Both of these are a ...
4
votes
1answer
319 views

Computationally Inexpensive Imputation Techniques in R

I have a large data-frame (155257 x 21 to be specific) with only a few missing values. Say, some 2.16% of the values need to be imputed. The values are floating point numbers. I'd like to use a ...
4
votes
2answers
1k views

Handling many missing values

I have a dataset I want to perform multivariate linear regression to it. The dimensions of the dataset are 832085 rows and 11 columns. The data are quite messy and given the size and my lack of ...
3
votes
2answers
679 views

How to handle missing data for machine learning

I'm trying to come up with a data structure to predict water visibility in a lake. I have some measured samples but would like to take other features into the equation. As an example, I would like to ...
3
votes
1answer
291 views

Predicting Missing Features

I have "millions" of items each having N binary features. When a feature is "0" it could be that the information is simply missing. So, given the data with the currently observed 1's, I would like to ...
3
votes
2answers
233 views

How does the naive Bayes classifier handle missing data in testing?

Assume that a classier has been trained already (no missing training data), but a prediction has been requested based on an observation that does not include every feature. How can we handle this ...
3
votes
1answer
240 views

Handling missing values to optimize polynomial features

I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on ...
3
votes
1answer
88 views

Correlation with missing values. Is least squares an acceptable option?

I have been tasked with finding a correlation matrix for a lot of variables. Many of them have missing values. I read here that pairwise deletion may not be the best way of dealing with this situation,...
3
votes
1answer
132 views

How to fix inconsistent (variable spelling) categorical data and “fill in” missing data

I am a newbie data science engr. My first challenge is to (1) normalize inconsistent values in categorical features and (2) fill any missing information. To describe inconsistency lets say we have a ...
3
votes
2answers
112 views

What is the difference between Missing at Random and Missing not at Random data?

I have been working with a dataset where the missing data seem to following a few particular patterns. I have gone through a lot websites and articles related to missing data but I haven't been able ...
3
votes
1answer
61 views

Setting “missing” distance values to zero when training a neural network

Not sure if missing values is the right name to use here. I want to train a DNN on data given by a sensor. The sensor gives the (x,y) coordinates of the founded objects. The sensor can keep track of ...
3
votes
1answer
25 views

How to deal with Missing Not at Random Data for k-means clustering?

I am running k-means clustering on a customer dataset. One of the available demographic fields is inferred homevalue, represented as an integer. This field has value 0 when it's inferred that the ...
2
votes
3answers
186 views

Cluster analysis as an associative model?

I have a set of data with many samples and many features, but where half of the data is missing one variable (call it A), which is composed of four categories. Based on the half of data which has A, I ...
2
votes
1answer
164 views

Data prepration for logistic regression : Value either “not available” or a “year”

I have some data of houses that have been renovated. In my data there is one column (among others) that captures this information. It is either "-1" if there has not been yet any renovation, or the ...
2
votes
1answer
516 views

Missing outputs in multiple-output neural net

I am looking at a task, where I want to predict multiple things from an image (an animal's breed [categorical], age [continuous number] and gender [categorical]). Unsurprisingly, my first thought was ...
2
votes
2answers
3k views

Methodologies for predicting missing data

I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset. For example: I have a dataset which contains information about a person. This can be ...
2
votes
1answer
265 views

How to treat missing data for survival analysis

I have a dataset consisting of questionnaires from patient survey data. There are around 10 questions which are asked during several stages of treatment like during first day of visit, after a week, ...
2
votes
1answer
234 views

Replacing missing value by class conditional mean

I have two classes, $p(x|y=0)$ and $p(x|y=1)$ with ${{\mu }_{0}}$ and ${{\mu }_{1}}$ as mean and shared covariance matrix $\Sigma $. Now, I have a missing feature ${{x}_{n}}$ for a particular ...
2
votes
4answers
2k views

Scikit Learn Missing Data - Categorical values

I have a dataset containing categorical features, which has 4 labels, and 4 features. (It is a meta classifier, so outputs from base classifier serve as input into this classifier) ...
2
votes
1answer
374 views

How to fill in missing value of the mean of the other columns?

I had a movie dataset including 'budget' and 'genres' attributes. I'd like to fill in the missing value of budget with the mean budget of each genre. I first create two dataframes with or without ...
2
votes
2answers
2k views

How can I handle missing categorical data that has significance?

I have a data set that is highly categorical and has a lot of missing values. For instance: ...
2
votes
1answer
24 views

Training on data with inherently non-applicable data cells

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense. Chemical Dataset Contains thousands of rows of chemical mixtures with many columns of ...
2
votes
2answers
27 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
2
votes
1answer
192 views

Toolbox for handling NaNs in Python 2.7

Is there a good toolbox for handling and analyzing missing values in Python 2.7? There is a good toolbox for doing this in Python 3.6 here (missingno): https://github.com/ResidentMario/missingno I ...
2
votes
1answer
2k views

Missing data imputation with KNN

-1 down vote favorite I have a dataset including missing data for most of the variables. Assume the dataset is as follows: ...
2
votes
1answer
188 views

Missing Categorical Features - no imputation

I've been reading about how to approach missing categorical features in test data, and the most common approach is to use imputation - for example using the last known value or getting the majority ...
2
votes
2answers
251 views

Feature scaling data with missing values

am interested in doing some feature scaling to try and tease out something from my data (box plots by outcome show that the 25/50/75 quantiles are very similar; certain variables have more "outliers" ...
2
votes
1answer
29 views

Cluster Analysis - Comparing Same Individuals Clustered Across Different Datasets with different features

I have an interesting problem, and I think my Google is failing me since I can't find the same problem anywhere. I have a set of individuals. I have 4 different datasets, with (some) to (all) of ...
2
votes
0answers
30 views

Detect Missing Records in Dataset

I have a dataset that contains several measures from various widgets on a daily basis. While the widgets remain relatively stable over time, sometimes there are legitimate reasons for one to disappear ...
2
votes
1answer
238 views

How to deal with missing data for Bernoulli Naive Bayes?

I am dealing with a dataset of categorical data that looks like this: ...
2
votes
2answers
21 views

Dealing with diverse groups in regression

What happens if a certain dataset contains different "groups" that follow different linear models? For example, let's imagine that examining the scatterplot of a certain feature $x_i$ against $y$ we ...
1
vote
2answers
58 views

Column With Many Missing Values (36%)

Hello this is my first machine learning project, I got a dataset with 18.000 rows and I have a column with 4244 values missing. I don't know why the values are missing since when it's appropriate ...
1
vote
1answer
1k views

Fill missing values AND normalise

I have two columns of training data for a neural net which are missing values. (There are many other columns which aren't missing values.) For example ...
1
vote
2answers
266 views

How would one impute missing values for a Discrete variable?

How would one imputing missing values (without using the mode) for a discrete variable, e.g. a variable corresponding to a count.
1
vote
1answer
63 views

Data that's not missing is called…?

Is there a standard term for data that's not missing? I.e. is it called non-missing, present, or something else? Thanks!
1
vote
1answer
86 views

Using predictive modelling for temperature data set

I am absolutely new to this area of predictive modelling in data science. I am not able to understand how and what modelling techniques do we use? Does it depend on the data type? Does it depend on ...
1
vote
1answer
266 views
1
vote
1answer
53 views

R mice doesn't give a 'valid' sollution

EDITED: See below for additional information.. TL;DR: How can I add missing data in a dataset like the sample in a way that it doesn't deviate much from the original dataset. ORIGINAL: I have a ...
1
vote
1answer
394 views

Imputation missing values other than using Mean, Median in python

I heard that Mean, Median isn't the best way to impute the missing values, why would that be? In my scenario, I have data like this ...
1
vote
1answer
84 views

Does encoding missing data with fixed values help in classification?

I have a lot of missing values for some variables in my data (70-80%). I have seen some people deal with missing values this way: encode the variable with missing values as 0 or 1. Where 0 is the ...
1
vote
0answers
19 views

How to incorporate an attribute that only exists in some observations?

In a binary classification problem, some of my observations have an event that occurs. I can, obviously, add a 1/0 flag if the event occurs ("event_occurred" in the data below). However, my intuition ...
1
vote
0answers
12 views

Finding features for missing classes

I have some data with labels from n-1 classes. There is no datapoint with the n-th class in the data but we know that data from the n-th exist. How can we generate data from the missing n+1-the class? ...
1
vote
1answer
36 views

Handling NA Values in the Chicago Crime Rate data set

I am doing a little project on the Chicago Crime Rate data set and I noticed that there are over 600,000 NA values, primarily in the location fields. I feel that ...
1
vote
1answer
2k views

Dealing with NaN (missing) values for Logistic Regression- Best practices?

I am working with a data-set of patient information and trying to calculate the Propensity Score from the data using MATLAB. After removing features with many missing values, I am still left with ...
1
vote
0answers
40 views