Questions tagged [missing-data]

Missing data is a problem that arises in data science when some data contained in rows or columns may be missing or unavailable for some samples in a dataset.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

TensorFlow Lite for Microcontrollers: Predicting with missing values and categorical variables

I work on a project where sensor data have categorical variables and missing values. Preprocessing sensor data with, for example, tfp.sts.impute_missing_values and ...
1
vote
1answer
24 views

How to fill missing consumption data on time series?

I have a dataset that contains consumptions. These consumptions are measured every month. But some months are not measured. So the measured month after the unmeasured month is actually worth the sum ...
1
vote
0answers
19 views

Missing data for time-series forecasting/classification

I want to train a model to predict some customer behavior. I have two types of data that I have to deal with. My measurements are daily. The first type is regular measurements without any missing data,...
1
vote
0answers
25 views

Identify MCAR, MNAR and MAR in the data

If I have missing values in a dataset, I can't just blindly impute them with mean/median/mode or any other technique. I have to identify what kind of missing values they are, namely: MCAR (missing ...
0
votes
0answers
4 views

How can I assess feature importance when determining whether a missing data is MCAR or not?

I was reading some lecture notes on missing data and the author suggests the following approach to determine whether some varibale is missing completely at random (MCAR) or not: Supervised Learning ...
0
votes
0answers
24 views

Gradient boosting algorithms and filling categorical variables

I have house prices dataset Link on Kaggle and I am having some dilemma. Some categorical variables having explicit majority. If we look at MSZoning and SaleType columns, there is "RL" type ...
0
votes
1answer
20 views

How to deal with large proportion of missing values in categorical variable

I have a dataset of around 5,500 observations. One of the variables is Gender for which at least 25% of the observations are missing. Dropping the missing values ...
0
votes
0answers
11 views

Imputing missing data of an univariate Sample

I have given a set of values $$ (x_1,...,x_n) $$ these are results of a specified model that predicts defaults for credit risks. For some of the observations the in the model needed information is ...
1
vote
2answers
64 views

Dealing with high number of NAs in a classification problem

I am working on a classification problem. The dataset dimension is as 187,643 x 203. The first column contains class labels with no NA. The rest of dataset are frequency data and could be anything ...
1
vote
1answer
16 views

Why does isna()/isnull() function show different values compared to value_counts()?

I am quite new to Machine Learning/Data science so I started doing my first Kaggle competition problem from here: https://www.kaggle.com/c/house-prices-advanced-regression-techniques I was cleaning ...
1
vote
2answers
15 views

Dimensionality reduction of vectors with null values

I have vectors of same length where each entry can have the value 0, 1 or null. V = {[0,1,1,1,null,0], [null,1,0,null,0,1], ...} How can I perform a dimensionality ...
2
votes
1answer
315 views

How to impute missing text data?

Lets say I have a dataframe consisting of two text columns. By text, I mean the values in those columns are either sentences/paragraphs. In such a case, how do I handle missing 'NaN' values? If it ...
1
vote
3answers
67 views

How does missing data occur

I am new to ML. I found that one of the preprocessing steps is to handle missing data. My query is Is there a way to understand nature of missing data I can see that the mostly missing data is ...
0
votes
1answer
71 views

What is the Better approach to handle Missing Values?

I read up some books on missing values. They have mentioned that listwise deletion is the least preferred method even though the sample size maybe be large (Newman, D. A. 2014. Missing Data: Five ...
4
votes
5answers
154 views

Dropping columns or inputing numbers

After looking at the various different ways of inputting data to replace NaN in a dataset vs. dropping observations or columns based on a threshold, the right ...
1
vote
0answers
20 views

How to figure out what elements are missing from a set, based on other sets?

I would like to solve a problem where I have a set of sets of possible values, but some elements of some sets are corrupted/deleted, so I had to figure out what is the most probable candidate ...
1
vote
0answers
10 views

Does it make more sense to use central tendency methods on training and test sets separately?

I'll explain further, so I'm taking a data science course on cleaning and preparing data and I'm on the how to handle missing data section. So the question is essentially what you see above except it ...
0
votes
0answers
23 views

Filling in missing values for time series data

Suppose I have time series data. So 200 cells of data with each row representing a millisecond - although some rows skip certain milliseconds. For example, it goes from 6:22:37 PM to 6:22:39 PM to 6:...
0
votes
0answers
21 views

How to handle missing IP addresses for a ml model?

I am using netflow data for a ml project that includes IP addresses. But it seems there are many missing values in IP address column. One way is to discard those, but I am trying to retain as many ...
1
vote
0answers
24 views

Naive Bayes model - Missing output variable for specific sample(s)

With Naive Bayes model can we discard a training sample if the value of the sample's output variable is missing? And would that not make any difference to the parameter learning of the Naive Bayes ...
1
vote
0answers
46 views

Model Tree M5 - Robustness to Data Quality Issues

I am currently investigating the M5 tree algorithm by Quinlan(1992) link here: https://sci2s.ugr.es/keel/pdf/algorithm/congreso/1992-Quinlan-AI.pdf An example of a linear regression model of the ...
0
votes
0answers
52 views

How to fill missing latitude and longitude values in time series data?

I have time-series data like this: date longitude latitude 01/01/2010 -5.42766 107.5784 02/01/2010 -6.42728 104.5245 07/01/2010 -7.42702 105.5816 14/01/2010 -4.42728 99.57834 17/01/2010 -6.41523 ...
0
votes
1answer
10 views

ffill missing data based on the sort order of another column

I have a column with missing data that I need to imput. Column is called 'Bandwidth'. There is a relationship between the Bandwidth column, and another column called 'Age'. As Age increases, so ...
1
vote
2answers
33 views

Dimensionality reduction for feature extraction when missing some feature values

I have two questions: 1-Which method is appropriate for dimensionality reduction for feature extraction when missing some feature values? 2-Which textbook is the best source for the answer in (1)?
0
votes
1answer
19 views

Impute missing value: transpose or not?

I'm building a model that fills the missing values from a Dataframe that contains the number of visitors for different stores, each day: day store_a store_b store_c 2021-01-01 100 200 300 2021-01-...
0
votes
0answers
25 views

Handling Null values

I am trying to fit a RandomForest model for a binary classification dataset and I have some feature like, the sales for a particular store and yes/no information ...
2
votes
4answers
104 views

Are there ML Libs in Python robust to missing data?

So I was searching on how to handle missing data and came across this post from Machine Learning Mastery. This article states that some algorithms can be made robust to missing data, such as Naive ...
1
vote
0answers
25 views

Active learning with mixture model cluster assignments - am I injecting bias here?

Suppose I have a dataset of people's phone numbers and heights, and I'm interested in learning the parameters $p_{girl}$, $p_{boy}=1-p_{girl}$, $\mu_{boy}$, $\mu_{girl}$, and overall $\sigma$ ...
0
votes
1answer
201 views

Imputing missing value based on filtering result of another column [closed]

C1 C2 A x A y A z A x A NaN A x A x A x B y B y B z B y B NaN B y B x B x I have to impute missing values in C2 , the imputation should be such that if the missing values ...
0
votes
1answer
63 views

Error in lmest: missing data in the covariates affecting the initial probabilities are not allowed

I'm running an LM model using the LMest package available in R. The dataset contains NO missing values. pct_miss(df_long) [1] 0 n_miss(df_long) [1] 0 The lmest function with no covariates works fine....
2
votes
1answer
64 views

How to treat patients without events in time-to-event analysis?

I'm working with longitudinal data for a series of patients. Duration of followup on a patient-level is non-uniform. Patients can either experience a discrete event (e.g., a heart attack) or never ...
0
votes
0answers
33 views

How are missing values treated in XGB RF Classifier?

I was exploring Random Forest Classifier in XGBoost listed here : https://xgboost.readthedocs.io/en/latest/python/python_api.html I was wondering how the missing values will be handled in this ...
0
votes
2answers
46 views

Can I deal with a missing not at random column by creating a new column? (Feature engineering)

Task: Binary classification Example problem: Let's say we have two feature columns A and B. A has no nulls and is a binary column if a user completed an action (=1), 0 if they didn't. For all users ...
0
votes
0answers
26 views

Do I remove the missing values before performing univariate and bivariate analysis? Is there a general rule?

I have read the answer to this question, but it doesn't quite answer my question whether there is a general rule for dealing with this situation. When performing EDA I find missing values in the ...
0
votes
1answer
165 views

the missForest function in R doesn't work

I'm trying to use the function missForest() of the library 'missForest' but I always get the same error message. This is the code: libraries: ...
3
votes
2answers
469 views

Do i need to handle missing values before EDA?

I am working on a data set and there is an interesting column with missing values, but I don't want to discard the rows (so as not to lose data from other columns) or do imputation (so as not to ...
1
vote
0answers
20 views

Handling missing data - secondary driver characteristics in insurance data

I have an insurance dataset which includes an indicator that indicates whether the policy insures a secondary driver, and the secondary driver's age/sex. Problem is most policies do not have secondary ...
4
votes
1answer
116 views

Why removing rows with NA values from the majority class improves model performance

I have an imbalanced dataset like so: df['y'].value_counts(normalize=True) * 100 ...
1
vote
1answer
37 views

How to deal with missing values in the survey data to perform Paired sample t test

I have a dataset where I have 100 respondents. Each respondent has to give response on service quality of Health care equipment. Is it providing efficient services to the patients? We have two ...
1
vote
1answer
261 views

Replace Missing Values with Most Frequent number under Condition

I'm trying to replace missing values of column "Age" but under condition of other columns on this data Titanic - Machine Learning from Disaster ...
3
votes
1answer
73 views

How data are prepared during training, testing and in production?

Most of real world datasets have features with missing values. Replacing missing values with an appropriate value such as its mean, is considered as a good step in feature engineering. Some times we ...
1
vote
1answer
31 views

Dealing with missing data in several features at once

Good day, What are the approaches for handling missing data in several features (categorical and continuous) at once? I look through each feature and plotted several histograms of the distribution of ...
1
vote
1answer
16 views

Dropping Missing Observations under MAR Assumption

Some of the outcome data in my data set are missing. I believe that the missing data mechanism is missing at random (MAR) as the observed characteristics significantly differ between the missing and ...
1
vote
2answers
107 views

How to build a model on a dataset having 40% missing values in most of the variables?

I have a huge dataset of 10 million observations but most of the variables are missing for 40% records. There are couple of variables available for the whole dataset such as sic code(Industry category)...
1
vote
1answer
34 views

How to handle a valuable feature that is missing on 99\% of the samples in the data set?

Suppose we have an input feature that is highly predictive of the outcome we want to predict. However, the feature is missing on 99% of the samples in the data set. What is the best way to use this ...
1
vote
0answers
31 views

How to handle large systematic missing data in time series?

I have this time series, where on the weekends, the dependent variable values are missing. It's only a time series, I do not have any exogenous regressors/features. The dependent variable value is an ...
0
votes
1answer
98 views

sklearn KNN imputation

Can I use sklearn's KNN imputer to fit the model to my training set and impute missing values in the test set using the neighbours from training set ? Is it allowed ? Or , Should I only fit and ...
1
vote
1answer
49 views

Dropping missing rows in two dataframes

I have two files : Test_data - contains the features of a dataset to find predictions for Submission_data - contains two columns : The index column for test data and another column for its ...
1
vote
1answer
91 views

Why we can't Remove features with missing values in Data Preprocessing [closed]

In a Real Time Dataset, There are many missing values available in the Dataset and also we need to deal with data preprocessing. And there are many ways to minimize the problem of missing values ...
0
votes
1answer
45 views

How to handle sparsely coded features in a dataframe

I have a dataset that contains information regarding diabetes patients, like so: ...