Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [missing-data]

The tag has no usage guidance.

0
votes
3answers
17 views

What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation. I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) ...
1
vote
0answers
44 views

How to deal with missing data for Bernoulli Naive Bayes?

I am dealing with a dataset of categorical data that looks like this: ...
0
votes
0answers
8 views

Encoding Target variable missing data imputing using KNN

I'm working on a classification model, so my target variable for the training set contains some missing data, I'm trying to impute it using KNN. Should I encode the target variable before training ...
1
vote
0answers
18 views

Handling missing values to optimize polynomial features

I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on ...
0
votes
1answer
27 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...
0
votes
1answer
33 views

Missing Values in Classification

I'm working on a classification problem. I'm trying to build a model which can predict if a bank client will get a loan or not. Some of clients have co-borrower and the majority don't. I also have ...
0
votes
1answer
140 views

To calculate unaffected part of the data set with missing values and positive skewness

A dataset has some missing values with positive skewness = 1. It is known that it is spread over 1.5 standard deviation from the median. How much % of data will remain unaffected?
1
vote
1answer
68 views

Dealing with NaN (missing) values for Logistic Regression- Best practices?

I am working with a data-set of patient information and trying to calculate the Propensity Score from the data using MATLAB. After removing features with many missing values, I am still left with ...
1
vote
1answer
31 views

How to treat missing data for survival analysis

I have a dataset consisting of questionnaires from patient survey data. There are around 10 questions which are asked during several stages of treatment like during first day of visit, after a week, ...
1
vote
2answers
16 views

Dealing with diverse groups in regression

What happens if a certain dataset contains different "groups" that follow different linear models? For example, let's imagine that examining the scatterplot of a certain feature $x_i$ against $y$ we ...
2
votes
2answers
35 views

What is the difference between Missing at Random and Missing not at Random data?

I have been working with a dataset where the missing data seem to following a few particular patterns. I have gone through a lot websites and articles related to missing data but I haven't been able ...
0
votes
1answer
23 views

what to do if the missing data in one column is based on some value/condition in another column in r?

I have a dataset with 20,000 observations and 19 variables. To start off with I have a gender column which has three levels namely 'M', 'F' and 'U' where U can be taken as not disclosed. Whenever ...
1
vote
1answer
28 views

Imputation missing values other than using Mean, Median in python

I heard that Mean, Median isn't the best way to impute the missing values, why would that be? In my scenario, I have data like this ...
3
votes
1answer
21 views

How to deal with Missing Not at Random Data for k-means clustering?

I am running k-means clustering on a customer dataset. One of the available demographic fields is inferred homevalue, represented as an integer. This field has value 0 when it's inferred that the ...
0
votes
1answer
32 views

Orange imports numeric features as categorical (“file” widget)

Why are some of my numeric features not being recognized as 'numeric' types AND why can't I reclassify them? I can't share my CSV here but I can assure you those features are indeed numeric (I use ...
1
vote
1answer
31 views

Missing value in continuous variable: Indicator variable vs. Indicator value

Most data has missing values, and as far as I'm aware, these are the options: Imputation (mean, hot-deck, etc.) Indicator variable. A categorical variable that tells what type the primary variable is....
0
votes
1answer
93 views

Fill missing values AND normalise

I have two columns of training data for a neural net which are missing values. (There are many other columns which aren't missing values.) For example ...
0
votes
0answers
28 views

scaling features after imputing missing values

I have a set of features with missing values. After imputing them by the median, I perform feature scaling (in Python/scikit i use preprocessing.scale and imputer). Now, there are many zeros which ...
2
votes
1answer
43 views

Toolbox for handling NaNs in Python 2.7

Is there a good toolbox for handling and analyzing missing values in Python 2.7? There is a good toolbox for doing this in Python 3.6 here (missingno): https://github.com/ResidentMario/missingno I ...
1
vote
0answers
17 views

Linear Model: How to deal with predictors with a lot of missing/small values?

I have a linear model used for prediction, with around 30 predictors, which are car usage rate as in percentage, across different zip codes. All these predictors have the same unit, as they are all ...
1
vote
1answer
258 views

Missing outputs in multiple-output neural net

I am looking at a task, where I want to predict multiple things from an image (an animal's breed [categorical], age [continuous number] and gender [categorical]). Unsurprisingly, my first thought was ...
0
votes
0answers
17 views

Fill missing continuous values using Machine Learning models

Is it a good idea to fill missing data with ML models, like Linear Regression or KNN, that use other highly correlated features to make predictions?
3
votes
1answer
57 views

Setting “missing” distance values to zero when training a neural network

Not sure if missing values is the right name to use here. I want to train a DNN on data given by a sensor. The sensor gives the (x,y) coordinates of the founded objects. The sensor can keep track of ...
0
votes
0answers
33 views

Scaling data with multiple Imputation with KNN?

Since it is required that data is normalised prior to imputing with KNN, if we were to scale, wouldn't this affect "imputation accuracy" ? If it would, what would you suggest in terms of an approach ...
1
vote
2answers
94 views

How would one impute missing values for a Discrete variable?

How would one imputing missing values (without using the mode) for a discrete variable, e.g. a variable corresponding to a count.
1
vote
1answer
77 views

Impute missing population values in Census data

I have population data from Census.gov: Total US population by age by year from 1940 through 2010 Depending on the range of decades, the data is missing discrete population values for ages greater ...
0
votes
0answers
309 views

How to handle missing date and time data

I have been working on handling missing data for a few days now but I have failed to come across good articles or content on handling missing dates, time or datetime objects. The normal methods of ...
1
vote
0answers
36 views

Fix missing data by adding another feature instead of using the mean?

I am trying to build a model which predicts whether a user will unsubscribe from a service. There is a particular column which tells the number of hours until a report was written for the user. These ...
3
votes
1answer
165 views

Predicting Missing Features

I have "millions" of items each having N binary features. When a feature is "0" it could be that the information is simply missing. So, given the data with the currently observed 1's, I would like to ...
0
votes
0answers
23 views

Computing regression with missing values

I'm to analyze results of education feedback questionnaire: program has many courses, and almost nobody takes every single one of them. My task is to predict program grade (made by students) based on ...
3
votes
1answer
48 views

Correlation with missing values. Is least squares an acceptable option?

I have been tasked with finding a correlation matrix for a lot of variables. Many of them have missing values. I read here that pairwise deletion may not be the best way of dealing with this situation,...
1
vote
0answers
344 views

predict() returns NA values

I have the problem. predict() method returns NA. My plan is: Read data from file and separate data to 2 sets: test and train Remove column with NA fraction over 95% Replace NA values with mean value ...
0
votes
1answer
34 views

Can we look just at the other features when we have a missing vaue?

For my classification task i have 3 features a,b,c. Feature c is missing for some datas. I can have already a good score for my classifier training with the two other features a,b and even better if I ...
0
votes
2answers
271 views

Filling missing values for important features [closed]

So I have this dataset from census the census bureau database which contains 40 attributes and a target column specifying if the total income is >50k. P.S : this is not the known ADULT dataset (which ...
0
votes
1answer
657 views

Filling missing values with pyspark using a probability distribution

I want to fill missing values in my dataframe. ...
-1
votes
1answer
202 views

What are the best way to handle missing values [closed]

Suppose we have a dataframe df in python, with numerical and categorical variables. For Numerical, when do we replace by mean and when by median. For ...
2
votes
1answer
171 views

Replacing missing value by class conditional mean

I have two classes, $p(x|y=0)$ and $p(x|y=1)$ with ${{\mu }_{0}}$ and ${{\mu }_{1}}$ as mean and shared covariance matrix $\Sigma $. Now, I have a missing feature ${{x}_{n}}$ for a particular ...
5
votes
4answers
1k views

Missing Values in Data [duplicate]

I have experienced that most of the datasets contain missing values, which make our task bit challenging. Please let me know how to fill up those missing values in an efficient way? and is there any ...
0
votes
0answers
42 views

XGBOOST missing_value feature degrades my performance?

I'm training a XGBOOST model for gout disease on a training set I sampled 1-to-7 case-control ratio (enriched in cases). I have 220 features and I reach a cross-...
4
votes
2answers
556 views

How to handle missing data for machine learning

I'm trying to come up with a data structure to predict water visibility in a lake. I have some measured samples but would like to take other features into the equation. As an example, I would like to ...
1
vote
1answer
64 views

Does encoding missing values help?

I have a lot of missing values for some variables in my data(70-80%) . I have seen some people deal with missing values this way : Encode the variable with missing values as 0 or 1. Where 0 is the ...
0
votes
2answers
331 views

how to do the imputation for categorical feature with a missing rate?

I have a dataset containing a categorical feature with a missing rate 95%. What value can replace the missing cells? Or drop this feature?
1
vote
1answer
493 views

How can I perform multi-label classification if many labels are missing? [closed]

I have a large set of documents, usually 500-2,000 words each, and for several different labels, there are about 20-100 samples with those labels, and hundreds to millions more that should be labeled ...
2
votes
1answer
1k views

Missing data imputation with KNN

-1 down vote favorite I have a dataset including missing data for most of the variables. Assume the dataset is as follows: ...
2
votes
2answers
652 views

How can I handle missing categorical data that has significance?

I have a data set that is highly categorical and has a lot of missing values. For instance: ...
0
votes
1answer
103 views

Choice of replacing missing values based on the data distribution

I am building a classification model based on a relatively small dataset. I have some missing values on the different attributes that I have. I cannot afford deleting any of the record that has ...
1
vote
2answers
2k views

Methodologies for predicting missing data

I have the following problem: I'm searching for methods to predict randomly missing data in a given dataset. For example: I have a dataset which contains information about a person. This can be ...
0
votes
4answers
82 views

Deploying the prediction model under missing values for test data

I have successfully built a logistic regression prediction model based on data set that is complete and clean, i.e., there is no missing values and the data is consistent. Now, for deploying the model ...