2022 Developer Survey is open! Take survey.

Questions tagged [missing-data]

Missing data is a problem that arises in data science when some data contained in rows or columns may be missing or unavailable for some samples in a dataset.

Filter by
Sorted by
Tagged with
0 votes
1 answer
31 views

How to compare between two methods of multivariate to filling NA

In the Titanic dataset, I performed two methods to fill Age NA. The first one is regression using Lasso: ...
user avatar
0 votes
2 answers
9 views

Correlations with NA or with Zeros?

When calculating correlations in R e.g. via cor is it better to treat missing data as NAs or as ...
user avatar
  • 480
0 votes
0 answers
16 views

Can anyone tell me why is my pipeline wrong?

I am trying to build a pipeline in order to perform GridSearchCV to find the best parameters. I already split the data into train and validation and have the following code: ...
user avatar
0 votes
0 answers
9 views

Filling NaN values

According to my knowledge, before filling nan values we have to check whether data is missing because of MCAR, MAR or MNAR and it depends on how features are correlated with each other and then make a ...
user avatar
1 vote
0 answers
19 views

How to deal with feature with different sample size?

I got a dataset that contains 50 features starting from 2009 to 2018. But one of the feature was only availiable since 2015 and unable to recover. I am concerning about if I train a model on the whole ...
user avatar
  • 9
1 vote
1 answer
60 views

How to deal with missing values that are supposed to be missing?

I am trying to predict loan defaults with a fairly moderate-sized dataset. I will probably be using logistic regression and random forest. I have around 35 variables and one of them classifies the ...
user avatar
  • 13
0 votes
0 answers
6 views

Handling missing values in IP addresses and key-like features

I have a log dataset that contains +30 features. One group of these features are of the following type, for example, request_id, user_partyrole_id, authentication_id, user_login_key and such ip and ...
user avatar
0 votes
1 answer
28 views

How do outliers and missing values impact these classifiers?

I am currently working with a bunch of classification models especially Logistic regression, KNN, Naive Bayes, SVM, and Decision Trees for my machine learning class. I know how to handle finding and ...
user avatar
0 votes
1 answer
17 views

Optimization of a simple M x N dataset

I have a dataset consisting of M questionnaires and N students. Each students replied to some questionnaires. I would like to make the dataset better, by removing some questionnaires and/or some ...
user avatar
0 votes
0 answers
11 views

How to deal with data that is only available for a certain category

I am working on a house pricer model, and I have a feature with values 0 or 1 to indicate if the rent price is capped by the government or not (houses with capped rents sell for much lower on average)....
user avatar
1 vote
1 answer
15 views

What to do when you are building a feature and the denominator is zero?

This is something that looks very simple to solve, but I couldn't find any hint - perhaps I'm not asking Google the right question. Let's say you own an Internet Company. You have the total ...
user avatar
0 votes
0 answers
12 views

Time Series Data Missing Value Treatment

I have an hourly time series data for a solar plant which covers 3 years (2019, 2020, 2021). I have a categorical feature named WWCode which has 54 unique values. WWCode is actually a weather ...
user avatar
0 votes
1 answer
21 views

Handling missing values in medical data

I have a medical dataset that contains maternal and foetal data during pregnancy. There are some missing values in the dataset that I am unsure how to handle. Here is a short example of my dataset: <...
user avatar
  • 323
1 vote
1 answer
17 views

How to fill missing values in a discrete column in sales predictions for a drug supply chain company

I have been working on a dataset that has data from a famous drug supply chain company. The first few records of the dataset look like the following; Another data accompanies this (primary) dataset. ...
user avatar
0 votes
1 answer
31 views

Outlier detection - missing values

I have a data science challenge in which two datasets are provided, the first one contains weather data (temperature, wind speed, and precipitation) for a number of days, and the other contains flight ...
user avatar
  • 1
1 vote
0 answers
6 views

Comparing two models with different (naive) baseline

I would like to compare a model with listwise deletion to a model with multiple imputation. However, the model with listwise deletion has a majority class of 70%, while the model with multiple ...
user avatar
  • 11
1 vote
2 answers
26 views

Handling missing data for the majority class

I'm working with an unbalanced (10:1) dataset for classification. I also have a bunch of missing data on certain columns. If I discard them all, I still have a 5:1 ratio, so I guess I can afford to ...
user avatar
0 votes
0 answers
9 views

How to use K-NN imputer without replacing with decimal values example ( 0.75,0.6) instead of binary outcome (yes or no, 1 or 0)?

I am trying to impute some missing categorical values using K-NN imputer, after imputation the missing values are replaced with some decimal numbers. I want to use K-NN as classifier and the output (...
user avatar
1 vote
0 answers
40 views

How does Kalman filtering work backwards?

For time series missing value imputation I am using the Kalman filter/smoothing approach, given in the imputeTS package. As Kalman filter is iterative and needs a view data points to make its estimate ...
user avatar
0 votes
0 answers
10 views

TensorFlow Lite for Microcontrollers: Predicting with missing values and categorical variables

I work on a project where sensor data have categorical variables and missing values. Preprocessing sensor data with, for example, tfp.sts.impute_missing_values and ...
user avatar
  • 263
1 vote
1 answer
28 views

How to fill missing consumption data on time series?

I have a dataset that contains consumptions. These consumptions are measured every month. But some months are not measured. So the measured month after the unmeasured month is actually worth the sum ...
user avatar
1 vote
0 answers
20 views

Missing data for time-series forecasting/classification

I want to train a model to predict some customer behavior. I have two types of data that I have to deal with. My measurements are daily. The first type is regular measurements without any missing data,...
user avatar
2 votes
0 answers
169 views

Identify MCAR, MNAR and MAR in the data

If I have missing values in a dataset, I can't just blindly impute them with mean/median/mode or any other technique. I have to identify what kind of missing values they are, namely: MCAR (missing ...
user avatar
  • 1,241
0 votes
0 answers
6 views

How can I assess feature importance when determining whether a missing data is MCAR or not?

I was reading some lecture notes on missing data and the author suggests the following approach to determine whether some varibale is missing completely at random (MCAR) or not: Supervised Learning ...
user avatar
0 votes
0 answers
27 views

Gradient boosting algorithms and filling categorical variables

I have house prices dataset Link on Kaggle and I am having some dilemma. Some categorical variables having explicit majority. If we look at MSZoning and SaleType columns, there is "RL" type ...
user avatar
0 votes
1 answer
36 views

How to deal with large proportion of missing values in categorical variable

I have a dataset of around 5,500 observations. One of the variables is Gender for which at least 25% of the observations are missing. Dropping the missing values ...
user avatar
0 votes
0 answers
13 views

Imputing missing data of an univariate Sample

I have given a set of values $$ (x_1,...,x_n) $$ these are results of a specified model that predicts defaults for credit risks. For some of the observations the in the model needed information is ...
user avatar
1 vote
2 answers
94 views

Dealing with high number of NAs in a classification problem

I am working on a classification problem. The dataset dimension is as 187,643 x 203. The first column contains class labels with no NA. The rest of dataset are frequency data and could be anything ...
user avatar
  • 11
1 vote
1 answer
24 views

Why does isna()/isnull() function show different values compared to value_counts()?

I am quite new to Machine Learning/Data science so I started doing my first Kaggle competition problem from here: https://www.kaggle.com/c/house-prices-advanced-regression-techniques I was cleaning ...
user avatar
1 vote
2 answers
32 views

Dimensionality reduction of vectors with null values

I have vectors of same length where each entry can have the value 0, 1 or null. V = {[0,1,1,1,null,0], [null,1,0,null,0,1], ...} How can I perform a dimensionality ...
user avatar
  • 111
2 votes
1 answer
776 views

How to impute missing text data?

Lets say I have a dataframe consisting of two text columns. By text, I mean the values in those columns are either sentences/paragraphs. In such a case, how do I handle missing 'NaN' values? If it ...
user avatar
1 vote
3 answers
91 views

How does missing data occur

I am new to ML. I found that one of the preprocessing steps is to handle missing data. My query is Is there a way to understand nature of missing data I can see that the mostly missing data is ...
user avatar
0 votes
0 answers
85 views

What is the Better approach to handle Missing Values?

I read up some books on missing values. They have mentioned that listwise deletion is the least preferred method even though the sample size maybe be large (Newman, D. A. 2014. Missing Data: Five ...
user avatar
  • 103
3 votes
4 answers
175 views

Dropping columns or inputing numbers

After looking at the various different ways of inputting data to replace NaN in a dataset vs. dropping observations or columns based on a threshold, the right ...
user avatar
1 vote
0 answers
22 views

How to figure out what elements are missing from a set, based on other sets?

I would like to solve a problem where I have a set of sets of possible values, but some elements of some sets are corrupted/deleted, so I had to figure out what is the most probable candidate ...
user avatar
  • 867
1 vote
0 answers
12 views

Does it make more sense to use central tendency methods on training and test sets separately?

I'll explain further, so I'm taking a data science course on cleaning and preparing data and I'm on the how to handle missing data section. So the question is essentially what you see above except it ...
user avatar
0 votes
0 answers
29 views

Filling in missing values for time series data

Suppose I have time series data. So 200 cells of data with each row representing a millisecond - although some rows skip certain milliseconds. For example, it goes from 6:22:37 PM to 6:22:39 PM to 6:...
user avatar
0 votes
0 answers
26 views

How to handle missing IP addresses for a ml model?

I am using netflow data for a ml project that includes IP addresses. But it seems there are many missing values in IP address column. One way is to discard those, but I am trying to retain as many ...
user avatar
  • 1,057
1 vote
0 answers
44 views

Naive Bayes model - Missing output variable for specific sample(s)

With Naive Bayes model can we discard a training sample if the value of the sample's output variable is missing? And would that not make any difference to the parameter learning of the Naive Bayes ...
user avatar
  • 11
1 vote
0 answers
57 views

Model Tree M5 - Robustness to Data Quality Issues

I am currently investigating the M5 tree algorithm by Quinlan(1992) link here: https://sci2s.ugr.es/keel/pdf/algorithm/congreso/1992-Quinlan-AI.pdf An example of a linear regression model of the ...
user avatar
0 votes
1 answer
12 views

ffill missing data based on the sort order of another column

I have a column with missing data that I need to imput. Column is called 'Bandwidth'. There is a relationship between the Bandwidth column, and another column called 'Age'. As Age increases, so ...
user avatar
0 votes
2 answers
36 views

Dimensionality reduction for feature extraction when missing some feature values

I have two questions: 1-Which method is appropriate for dimensionality reduction for feature extraction when missing some feature values? 2-Which textbook is the best source for the answer in (1)?
user avatar
0 votes
1 answer
25 views

Impute missing value: transpose or not?

I'm building a model that fills the missing values from a Dataframe that contains the number of visitors for different stores, each day: day store_a store_b store_c 2021-01-01 100 200 300 2021-01-...
user avatar
0 votes
0 answers
36 views

Handling Null values

I am trying to fit a RandomForest model for a binary classification dataset and I have some feature like, the sales for a particular store and yes/no information ...
user avatar
  • 179
2 votes
4 answers
194 views

Are there ML Libs in Python robust to missing data?

So I was searching on how to handle missing data and came across this post from Machine Learning Mastery. This article states that some algorithms can be made robust to missing data, such as Naive ...
user avatar
1 vote
0 answers
25 views

Active learning with mixture model cluster assignments - am I injecting bias here?

Suppose I have a dataset of people's phone numbers and heights, and I'm interested in learning the parameters $p_{girl}$, $p_{boy}=1-p_{girl}$, $\mu_{boy}$, $\mu_{girl}$, and overall $\sigma$ ...
user avatar
  • 76
0 votes
1 answer
402 views

Imputing missing value based on filtering result of another column [closed]

C1 C2 A x A y A z A x A NaN A x A x A x B y B y B z B y B NaN B y B x B x I have to impute missing values in C2 , the imputation should be such that if the missing values ...
user avatar
  • 48
0 votes
1 answer
96 views

Error in lmest: missing data in the covariates affecting the initial probabilities are not allowed

I'm running an LM model using the LMest package available in R. The dataset contains NO missing values. pct_miss(df_long) [1] 0 n_miss(df_long) [1] 0 The lmest function with no covariates works fine....
user avatar
  • 1
2 votes
1 answer
64 views

How to treat patients without events in time-to-event analysis?

I'm working with longitudinal data for a series of patients. Duration of followup on a patient-level is non-uniform. Patients can either experience a discrete event (e.g., a heart attack) or never ...
user avatar
0 votes
2 answers
49 views

Can I deal with a missing not at random column by creating a new column? (Feature engineering)

Task: Binary classification Example problem: Let's say we have two feature columns A and B. A has no nulls and is a binary column if a user completed an action (=1), 0 if they didn't. For all users ...
user avatar