Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [missing-data]

The tag has no usage guidance.

0
votes
0answers
6 views

How to handle NULLS in a column with scalar values (for neural networks)

I'm hoping this is the correct location for this question and apologies if not. I have a data table of customer information vs an outcome. Each row represents a product purchase, and the product ...
0
votes
0answers
6 views

Fill missing values in a graph

Having a graph with attributes in the edeges that in some cases are missing, and asuming that those missing values have a releation with neighbors edges, how can I deduce them according different ...
0
votes
0answers
5 views

Weakly supervised learning and missing labels for data that likely contains that label

Cross-post from Cross Validated, because here seems more approperiate. I would like to know how to deal with data that misses a label, but is likely to contain the label in a weakly supervised ...
2
votes
2answers
21 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
0
votes
1answer
16 views

Handling NA Values in the Chicago Crime Rate data set

I am doing a little project on the Chicago Crime Rate data set and I noticed that there are over 600,000 NA values, primarily in the location fields. I feel that ...
1
vote
1answer
48 views

R mice doesn't give a 'valid' sollution

EDITED: See below for additional information.. TL;DR: How can I add missing data in a dataset like the sample in a way that it doesn't deviate much from the original dataset. ORIGINAL: I have a ...
2
votes
0answers
24 views

Detect Missing Records in Dataset

I have a dataset that contains several measures from various widgets on a daily basis. While the widgets remain relatively stable over time, sometimes there are legitimate reasons for one to disappear ...
0
votes
3answers
24 views

What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation. I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) ...
1
vote
0answers
92 views

How to deal with missing data for Bernoulli Naive Bayes?

I am dealing with a dataset of categorical data that looks like this: ...
0
votes
0answers
19 views

Encoding Target variable missing data imputing using KNN

I'm working on a classification model, so my target variable for the training set contains some missing data, I'm trying to impute it using KNN. Should I encode the target variable before training ...
1
vote
0answers
43 views

Handling missing values to optimize polynomial features

I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on ...
0
votes
1answer
27 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...
0
votes
1answer
36 views

Missing Values in Classification

I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't. I also have ...
0
votes
1answer
372 views

To calculate unaffected part of the data set with missing values and positive skewness

A dataset has some missing values with positive skewness = 1. It is known that it is spread over 1.5 standard deviation from the median. How much % of data will remain unaffected?
1
vote
1answer
270 views

Dealing with NaN (missing) values for Logistic Regression- Best practices?

I am working with a data-set of patient information and trying to calculate the Propensity Score from the data using MATLAB. After removing features with many missing values, I am still left with ...
2
votes
1answer
75 views

How to treat missing data for survival analysis

I have a dataset consisting of questionnaires from patient survey data. There are around 10 questions which are asked during several stages of treatment like during first day of visit, after a week, ...
2
votes
2answers
18 views

Dealing with diverse groups in regression

What happens if a certain dataset contains different "groups" that follow different linear models? For example, let's imagine that examining the scatterplot of a certain feature $x_i$ against $y$ we ...
2
votes
2answers
52 views

What is the difference between Missing at Random and Missing not at Random data?

I have been working with a dataset where the missing data seem to following a few particular patterns. I have gone through a lot websites and articles related to missing data but I haven't been able ...
0
votes
1answer
30 views

what to do if the missing data in one column is based on some value/condition in another column in r?

I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever ...
1
vote
1answer
60 views

Imputation missing values other than using Mean, Median in python

I heard that Mean, Median isn't the best way to impute the missing values, why would that be? In my scenario, I have data like this ...
3
votes
1answer
23 views

How to deal with Missing Not at Random Data for k-means clustering?

I am running k-means clustering on a customer dataset. One of the available demographic fields is inferred homevalue, represented as an integer. This field has value 0 when it's inferred that the ...
0
votes
1answer
58 views

Orange imports numeric features as categorical (“file” widget)

Why are some of my numeric features not being recognized as 'numeric' types AND why can't I reclassify them? I can't share my CSV here but I can assure you those features are indeed numeric (I use ...
1
vote
1answer
37 views

Missing value in continuous variable: Indicator variable vs. Indicator value

Most data has missing values, and as far as I'm aware, these are the options: Imputation (mean, hot-deck, etc.) Indicator variable. A categorical variable that tells what type the primary variable is....
0
votes
1answer
321 views

Fill missing values AND normalise

I have two columns of training data for a neural net which are missing values. (There are many other columns which aren't missing values.) For example ...
0
votes
0answers
35 views

scaling features after imputing missing values

I have a set of features with missing values. After imputing them by the median, I perform feature scaling (in Python/scikit i use preprocessing.scale and imputer). Now, there are many zeros which ...
2
votes
1answer
66 views

Toolbox for handling NaNs in Python 2.7

Is there a good toolbox for handling and analyzing missing values in Python 2.7? There is a good toolbox for doing this in Python 3.6 here (missingno): https://github.com/ResidentMario/missingno I ...
1
vote
0answers
18 views

Linear Model: How to deal with predictors with a lot of missing/small values?

I have a linear model used for prediction, with around 30 predictors, which are car usage rate as in percentage, across different zip codes. All these predictors have the same unit, as they are all ...
1
vote
1answer
318 views

Missing outputs in multiple-output neural net

I am looking at a task, where I want to predict multiple things from an image (an animal's breed [categorical], age [continuous number] and gender [categorical]). Unsurprisingly, my first thought was ...
0
votes
0answers
17 views

Fill missing continuous values using Machine Learning models

Is it a good idea to fill missing data with ML models, like Linear Regression or KNN, that use other highly correlated features to make predictions?
3
votes
1answer
57 views

Setting “missing” distance values to zero when training a neural network

Not sure if missing values is the right name to use here. I want to train a DNN on data given by a sensor. The sensor gives the (x,y) coordinates of the founded objects. The sensor can keep track of ...
0
votes
0answers
42 views

Scaling data with multiple Imputation with KNN?

Since it is required that data is normalised prior to imputing with KNN, if we were to scale, wouldn't this affect "imputation accuracy" ? If it would, what would you suggest in terms of an approach ...
1
vote
2answers
149 views

How would one impute missing values for a Discrete variable?

How would one imputing missing values (without using the mode) for a discrete variable, e.g. a variable corresponding to a count.
1
vote
1answer
99 views

Missing population values in census data

I have population data from Census.gov: Total US population by age by year from 1940 through 2010 Depending on the range of decades, the data is missing discrete population values for ages greater ...
0
votes
0answers
378 views

How to handle missing date and time data

I have been working on handling missing data for a few days now but I have failed to come across good articles or content on handling missing dates, time or datetime objects. The normal methods of ...
1
vote
0answers
38 views

Fix missing data by adding another feature instead of using the mean?

I am trying to build a model which predicts whether a user will unsubscribe from a service. There is a particular column which tells the number of hours until a report was written for the user. These ...
3
votes
1answer
195 views

Predicting Missing Features

I have "millions" of items each having N binary features. When a feature is "0" it could be that the information is simply missing. So, given the data with the currently observed 1's, I would like to ...
3
votes
1answer
62 views

Correlation with missing values. Is least squares an acceptable option?

I have been tasked with finding a correlation matrix for a lot of variables. Many of them have missing values. I read here that pairwise deletion may not be the best way of dealing with this situation,...
1
vote
0answers
398 views

predict() returns NA values

I have the problem. predict() method returns NA. My plan is: Read data from file and separate data to 2 sets: test and train Remove column with NA fraction over 95% Replace NA values with mean value ...
0
votes
1answer
34 views

Can we look just at the other features when we have a missing vaue?

For my classification task i have 3 features a,b,c. Feature c is missing for some datas. I can have already a good score for my classifier training with the two other features a,b and even better if I ...
0
votes
2answers
292 views

Filling missing values for important features [closed]

So I have this dataset from census the census bureau database which contains 40 attributes and a target column specifying if the total income is >50k. P.S : this is not the known ADULT dataset (which ...
0
votes
2answers
837 views

Filling missing values with pyspark using a probability distribution

I want to fill missing values in my dataframe. ...
-1
votes
1answer
235 views

What are the best way to handle missing values [closed]

Suppose we have a dataframe df in python, with numerical and categorical variables. For Numerical, when do we replace by mean and when by median. For ...
2
votes
1answer
198 views

Replacing missing value by class conditional mean

I have two classes, $p(x|y=0)$ and $p(x|y=1)$ with ${{\mu }_{0}}$ and ${{\mu }_{1}}$ as mean and shared covariance matrix $\Sigma $. Now, I have a missing feature ${{x}_{n}}$ for a particular ...
5
votes
4answers
2k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
0
votes
0answers
52 views

XGBOOST missing_value feature degrades my performance?

I'm training a XGBOOST model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-...
4
votes
2answers
591 views

How to handle missing data for machine learning

I'm trying to come up with a data structure to predict water visibility in a lake. I have some measured samples but would like to take other features into the equation. As an example, I would like to ...
1
vote
1answer
68 views

Does encoding missing values help?

I have a lot of missing values for some variables in my data(70-80%) . I have seen some people deal with missing values this way : Encode the variable with missing values as 0 or 1. Where 0 is the ...
0
votes
2answers
423 views

how to do the imputation for categorical feature with a missing rate?

I have a dataset containing a categorical feature with a missing rate 95%. What value can replace the missing cells? Or drop this feature?