Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [missing-data]

The tag has no usage guidance.

2
votes
1answer
157 views

Data prepration for logistic regression : Value either “not available” or a “year”

I have some data of houses that have been renovated. In my data there is one column (among others) that captures this information. It is either "-1" if there has not been yet any renovation, or the ...
0
votes
0answers
10 views

Learning with missing features (MNAR)

I want to learn from features that may have some missing informations. The value of the variable that's missing is related to the reason it's missing (MNAR) To better understand my case, here is an ...
2
votes
1answer
24 views

Cluster Analysis - Comparing Same Individuals Clustered Across Different Datasets with different features

I have an interesting problem, and I think my Google is failing me since I can't find the same problem anywhere. I have a set of individuals. I have 4 different datasets, with (some) to (all) of ...
1
vote
0answers
11 views

Finding features for missing classes

I have some data with labels from n-1 classes. There is no datapoint with the n-th class in the data but we know that data from the n-th exist. How can we generate data from the missing n+1-the class? ...
0
votes
0answers
9 views

How to impute opt out data

Consider the following problem: You have been given some survey data on job satisfaction and have to create a model for the employees with the highest risk of leaving the company. The questions have ...
2
votes
1answer
20 views

Training on data with inherently non-applicable data cells

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense. Chemical Dataset Contains thousands of rows of chemical mixtures with many columns of ...
0
votes
0answers
5 views

Calculating target mean to validate if I should drop column with missing values is correct?

I am working on the KDD 2009 Cup Data Set (The Small one) and I have a question about preprocessing data. It has a lot of columns with null values, some of them have more than 90% of missing. Reading ...
2
votes
1answer
44 views

How to fill in missing value of the mean of the other columns?

I had a movie dataset including 'budget' and 'genres' attributes. I'd like to fill in the missing value of budget with the mean budget of each genre. I first create two dataframes with or without ...
3
votes
2answers
84 views

How does the naive Bayes classifier handle missing data in testing?

Assume that a classier has been trained already (no missing training data), but a prediction has been requested based on an observation that does not include every feature. How can we handle this ...
0
votes
0answers
54 views

Finding unaffected data in a dataset

The data set has missing values. Further examination tells that they are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data (in percentage)...
-1
votes
0answers
10 views

How to handle NULLS in a column with scalar values (for neural networks)

I'm hoping this is the correct location for this question and apologies if not. I have a data table of customer information vs an outcome. Each row represents a product purchase, and the product ...
-1
votes
0answers
11 views

Fill missing values in a graph

Having a graph with attributes in the edeges that in some cases are missing, and asuming that those missing values have a releation with neighbors edges, how can I deduce them according different ...
-1
votes
0answers
13 views

Weakly supervised learning and missing labels for data that likely contains that label

Cross-post from Cross Validated, because here seems more approperiate. I would like to know how to deal with data that misses a label, but is likely to contain the label in a weakly supervised ...
2
votes
2answers
24 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
0
votes
1answer
20 views

Handling NA Values in the Chicago Crime Rate data set

I am doing a little project on the Chicago Crime Rate data set and I noticed that there are over 600,000 NA values, primarily in the location fields. I feel that ...
1
vote
1answer
49 views

R mice doesn't give a 'valid' sollution

EDITED: See below for additional information.. TL;DR: How can I add missing data in a dataset like the sample in a way that it doesn't deviate much from the original dataset. ORIGINAL: I have a ...
2
votes
0answers
24 views

Detect Missing Records in Dataset

I have a dataset that contains several measures from various widgets on a daily basis. While the widgets remain relatively stable over time, sometimes there are legitimate reasons for one to disappear ...
0
votes
3answers
30 views

What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation. I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) ...
1
vote
0answers
147 views

How to deal with missing data for Bernoulli Naive Bayes?

I am dealing with a dataset of categorical data that looks like this: ...
2
votes
0answers
85 views

Handling missing values to optimize polynomial features

I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on ...
0
votes
1answer
30 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...
0
votes
1answer
37 views

Missing Values in Classification

I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't. I also have ...
0
votes
1answer
1k views

To calculate unaffected part of the data set with missing values and positive skewness

A dataset has some missing values with positive skewness = 1. It is known that it is spread over 1.5 standard deviation from the median. How much % of data will remain unaffected?
1
vote
1answer
665 views

Dealing with NaN (missing) values for Logistic Regression- Best practices?

I am working with a data-set of patient information and trying to calculate the Propensity Score from the data using MATLAB. After removing features with many missing values, I am still left with ...
2
votes
1answer
154 views

How to treat missing data for survival analysis

I have a dataset consisting of questionnaires from patient survey data. There are around 10 questions which are asked during several stages of treatment like during first day of visit, after a week, ...
2
votes
2answers
19 views

Dealing with diverse groups in regression

What happens if a certain dataset contains different "groups" that follow different linear models? For example, let's imagine that examining the scatterplot of a certain feature $x_i$ against $y$ we ...
2
votes
2answers
72 views

What is the difference between Missing at Random and Missing not at Random data?

I have been working with a dataset where the missing data seem to following a few particular patterns. I have gone through a lot websites and articles related to missing data but I haven't been able ...
0
votes
1answer
31 views

what to do if the missing data in one column is based on some value/condition in another column in r?

I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever ...
1
vote
1answer
131 views

Imputation missing values other than using Mean, Median in python

I heard that Mean, Median isn't the best way to impute the missing values, why would that be? In my scenario, I have data like this ...
3
votes
1answer
24 views

How to deal with Missing Not at Random Data for k-means clustering?

I am running k-means clustering on a customer dataset. One of the available demographic fields is inferred homevalue, represented as an integer. This field has value 0 when it's inferred that the ...
0
votes
1answer
84 views

Orange imports numeric features as categorical (“file” widget)

Why are some of my numeric features not being recognized as 'numeric' types AND why can't I reclassify them? I can't share my CSV here but I can assure you those features are indeed numeric (I use ...
1
vote
1answer
40 views

Missing value in continuous variable: Indicator variable vs. Indicator value

Most data has missing values, and as far as I'm aware, these are the options: Imputation (mean, hot-deck, etc.) Indicator variable. A categorical variable that tells what type the primary variable is....
0
votes
1answer
604 views

Fill missing values AND normalise

I have two columns of training data for a neural net which are missing values. (There are many other columns which aren't missing values.) For example ...
2
votes
1answer
103 views

Toolbox for handling NaNs in Python 2.7

Is there a good toolbox for handling and analyzing missing values in Python 2.7? There is a good toolbox for doing this in Python 3.6 here (missingno): https://github.com/ResidentMario/missingno I ...
1
vote
0answers
18 views

Linear Model: How to deal with predictors with a lot of missing/small values?

I have a linear model used for prediction, with around 30 predictors, which are car usage rate as in percentage, across different zip codes. All these predictors have the same unit, as they are all ...
2
votes
1answer
392 views

Missing outputs in multiple-output neural net

I am looking at a task, where I want to predict multiple things from an image (an animal's breed [categorical], age [continuous number] and gender [categorical]). Unsurprisingly, my first thought was ...
3
votes
1answer
57 views

Setting “missing” distance values to zero when training a neural network

Not sure if missing values is the right name to use here. I want to train a DNN on data given by a sensor. The sensor gives the (x,y) coordinates of the founded objects. The sensor can keep track of ...
1
vote
2answers
203 views

How would one impute missing values for a Discrete variable?

How would one imputing missing values (without using the mode) for a discrete variable, e.g. a variable corresponding to a count.
1
vote
1answer
108 views

Missing population values in census data

I have population data from Census.gov: Total US population by age by year from 1940 through 2010 Depending on the range of decades, the data is missing discrete population values for ages greater ...
1
vote
0answers
39 views

Fix missing data by adding another feature instead of using the mean?

I am trying to build a model which predicts whether a user will unsubscribe from a service. There is a particular column which tells the number of hours until a report was written for the user. These ...
3
votes
1answer
232 views

Predicting Missing Features

I have "millions" of items each having N binary features. When a feature is "0" it could be that the information is simply missing. So, given the data with the currently observed 1's, I would like to ...
3
votes
1answer
73 views

Correlation with missing values. Is least squares an acceptable option?

I have been tasked with finding a correlation matrix for a lot of variables. Many of them have missing values. I read here that pairwise deletion may not be the best way of dealing with this situation,...
1
vote
0answers
466 views

predict() returns NA values

I have the problem. predict() method returns NA. My plan is: Read data from file and separate data to 2 sets: test and train Remove column with NA fraction over 95% Replace NA values with mean value ...
0
votes
1answer
34 views

Can we look just at the other features when we have a missing vaue?

For my classification task i have 3 features a,b,c. Feature c is missing for some datas. I can have already a good score for my classifier training with the two other features a,b and even better if I ...
0
votes
2answers
311 views

Filling missing values for important features [closed]

So I have this dataset from census the census bureau database which contains 40 attributes and a target column specifying if the total income is >50k. P.S : this is not the known ADULT dataset (which ...
0
votes
2answers
966 views

Filling missing values with pyspark using a probability distribution

I want to fill missing values in my dataframe. ...
-1
votes
1answer
263 views

What are the best way to handle missing values [closed]

Suppose we have a dataframe df in python, with numerical and categorical variables. For Numerical, when do we replace by mean and when by median. For ...
2
votes
1answer
214 views

Replacing missing value by class conditional mean

I have two classes, $p(x|y=0)$ and $p(x|y=1)$ with ${{\mu }_{0}}$ and ${{\mu }_{1}}$ as mean and shared covariance matrix $\Sigma $. Now, I have a missing feature ${{x}_{n}}$ for a particular ...