Questions tagged [missing-data]
Missing data is a problem that arises in data science when some data contained in rows or columns may be missing or unavailable for some samples in a dataset.
202
questions
0
votes
0
answers
41
views
Missing values handling in LightGBM
I'm a bit confused about the handling of missing data by LightGBM. I'm using the R package but my question should not be language-specific.
In a regression setting with no categorical feature, I have ...
- 101
0
votes
0
answers
7
views
How to selectively train a deep model based on the unavailability of a subset of the feature set
I am creating a deep learning binary classification model. Each sample in the dataset contains two mutually exclusive feature sets X and Y.
Feature set X is present in all samples; however, there are ...
0
votes
1
answer
18
views
Interpolating missing data given county and state totals
Problem:
I have population data for states and their constituent counties over several years. Each row is uniquely identified by a state/county and a year. There are four population columns: ...
0
votes
0
answers
41
views
Understanding the working of 'Interpolation'
'Interpolation' is a technique that is used to predict a null/empty value by studying its neighbouring points.
Interpolation doesn't take into consideration the entire dataset when predicting the ...
- 275
0
votes
1
answer
59
views
Dropping dataframe columns with missing values
When I looked up different sources that talk about dropping the columns of a dataframe that contains missing values, I got answers that are starkly different from one another.
Some sources say, ...
- 275
0
votes
2
answers
20
views
Correlation Analysis dealing with missing value, when the missing values are actually expected
So I am working on the correlation analysis in a dataset and trying to figure out the most sensible way to handle missing values. In my case, the missing values are expected.
To make it clearer, here ...
- 101
0
votes
0
answers
75
views
How to input or estimate missing text data?
My task requires 32 different columns with 25 beeing independent text data.
Deleting nan values, or cutting columns which has less then 20% (of non NaNs) results in reducing dataset to less then 10% ...
- 1
0
votes
2
answers
134
views
What is the state-of-the-art in prediction\classification missing labels in partially labeled data?
Overview
Let's say I have the following data:
...
- 364
0
votes
0
answers
31
views
Handling Missing Values
I have the following artificial data :
Now I am trying to handle missing values in the age and salary columns using mean imputation. I am using the following code to do so :
...
- 221
0
votes
1
answer
73
views
Is it essential to remove NULL values when doing EDA?
I'm doing a Data Analytics project on Youtube Statistics.
Assume that I want to get into the entertainment industry and decided to make a youtube channel. Before that, I need to identify what content ...
- 1
0
votes
0
answers
22
views
Interpolation techniques predicting negative values in a column with positive values
I tried different interpolation techniques & for some numeric columns with only positive values the interpolation techniques: Polynomial, Cubic Spline & Radial Basis are predicting negative ...
- 275
0
votes
1
answer
31
views
Best strategy for handling missing groups of features
I am currently working on a ML problem where the features used for modelling are sourced from different places/providers. It is very unlikely to find the features from all the different sources to be ...
0
votes
1
answer
84
views
How to handle date columns in a dataset with over 100+ date columns?
Background: I have a dataset with around 85k rows and 320 columns.I have no formal domain knowledge and the columns ain't intuitive as well the dataset is not in a language that I speak or understand. ...
0
votes
1
answer
110
views
Feature selection / missing values
What are the top (including new, if any) algorithms to perform features selections without removing or altering the missing data points ? Thanks
- 1
0
votes
1
answer
96
views
Splitting CSV data with missing target variable using train_test_split in Python
I have a dataset in CSV format that consists of a training dataset with around 300 instances and a test dataset with around 100 instances. The issue is that the target variable (the column we want to ...
0
votes
1
answer
49
views
How to treat missing values depending on what missing means
I have a dataset with quotes from an insurance company. I am trying to create a model to predict how much should the company charge the customer according to the different variables. Two of the ...
- 103
0
votes
1
answer
144
views
What's the best approach to dealing with missing data in a dataset?
I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing ...
- 1
0
votes
2
answers
30
views
Data imputation for heavily missing features
I am currently working on the dataset IEEE-CIS Fraud Detection, provided via Kaggle, with around 350 features, with around 600k instances. However, some features are missing large amounts of values, ...
0
votes
1
answer
34
views
important feature selection using dimensionality reduction algorithms
I have a dataset having more than 25000 features. I did perform noise removal using the histogram approach, and this dataset gets reduced to more than 5000 features. There are two classes, healthy and ...
0
votes
1
answer
23
views
Best practice for variables that only have answer if yes in previous column
I currently have a dataset that consists of survey data that has several columns that have answers dependent on the previous question. For example, I may have a question that says "Did you take ...
- 23
7
votes
5
answers
532
views
How to handle missing value if imputation doesnt make sense
I have column/feature in my dataset showing years a person has been married "years_married".
Since not every person is married there are NaN fields.
It does not make sense to fillna(0) "...
- 111
4
votes
3
answers
284
views
Imputation of missing values based on target variable
I want to impute missing values in German Credit Risk dataset.
...
- 61
2
votes
1
answer
193
views
Surrogate splits in Python
I want to use RandomForestClassifier from Sklearn to predict categorical variable (credit risk). But one of the predictors seems to have missing values:
...
- 61
0
votes
0
answers
41
views
Imputing Survival Analysis Data in R
The dataset contains 418 observations and 18 total variables. The two dependent variables are N_Days and Status. The dependent ...
- 3
0
votes
0
answers
10
views
How to approach a problem of identifying misclassified categories
So I have a dataset of where people have to classify repair cases narratives according to a dropdown. The dropdown has a default type and my boss informs me that an unknown number are probably ...
0
votes
0
answers
41
views
What are the advantages of keeping NAN values instead of handling/replacing them? Pandas
In Pandas, what are the benefits of keeping the nan values in the DataFrame/Series instead of replacing/handling them with another value? Can we use them to indicate an explicit state or flag? Are ...
- 101
0
votes
0
answers
14
views
Aggregating irregular recorded accelerometer and gps data
I have a dataset containing accelerometer- and GPS data on which I want to try some classification. However, I am not sure how to aggregate this data into a larger timescale (per day or per week) as ...
0
votes
1
answer
28
views
How to generate missing values for categorical attributes on a dataset?
I am working for my thesis on 3 known datasets adult,titanic and compass receidivism and i am trying to generate missing values for different missing rates on attributes(e.g Sex,Race) that contain ...
0
votes
1
answer
928
views
Trying to fill null values with sub-grouped mean value using pandas fillna() and groupby().transform() is doing nothing with the null values
I'm doing EDA on the US police shooting dataset where I'm stuck with filling the 482 null values for the age column. Since no strategy was mentioned on how to fill ...
- 111
0
votes
1
answer
158
views
How do I fill the null ages of a person according to the average age of males or females in a particular race
I got this US police shootings dataset where many age are null. Each person's race and gender...
- 111
1
vote
1
answer
31
views
Imputing missing values for "days ago" feature
I have a dataset with features such as last_visit_n_days_ago, last_purchase_n_days_ago. These features are unavailable for many ...
0
votes
1
answer
98
views
Handling NaN values in a feature representing prioritization ranking
I have a data set like so
...
- 131
0
votes
1
answer
44
views
Variable selection and NA
I have a very large dataset with a lot of NAs in the data. I want to perform an analysis and have to select the variables that are of most interest.
I feel like I have to take 3 steps before I can ...
- 13
0
votes
1
answer
53
views
Missing Standard Deviation Values
Currently trying to conduct a meta-analysis. I have reviewed the references I would like to use and made sure each reference met my criteria for this meta-analysis. However, majority of the references ...
- 1
1
vote
1
answer
60
views
Which is the right imputation method for stock data?
I am currently working on a project involving stock close prices of some companies. I encounter the problem of missing data in the dataset. For example in the image below, because, I have some null ...
- 11
1
vote
1
answer
33
views
Continuous variables with missing values of type MNAR
I have continuous variables (weather features) with missing values of type MNAR (a different distribution with and without missing values).
I learned that these variables should be transformed into ...
- 15
0
votes
2
answers
237
views
Missing Values Correlation
Is it worth it to study missingness correlation between columns?
If you have strongly correlated missing values (say between two columns, A and B), how will this change or shape the way you look at ...
1
vote
0
answers
91
views
Error from XGBoost missing data handling
I have a regression problem with a very large dataset >50 million rows, 81 features and 1 target, all positive float values unevenly distributed between 0 - 1 million. I've trained an XGBoost model ...
- 36
0
votes
1
answer
54
views
How to compare between two methods of multivariate to filling NA
In the Titanic dataset, I performed two methods to fill Age NA.
The first one is regression using Lasso:
...
0
votes
2
answers
220
views
Correlations with NA or with Zeros?
When calculating correlations in R e.g. via cor is it better to treat missing data as NAs or as ...
- 540
1
vote
1
answer
109
views
How to deal with missing values that are supposed to be missing?
I am trying to predict loan defaults with a fairly moderate-sized dataset. I will probably be using logistic regression and random forest.
I have around 35 variables and one of them classifies the ...
- 13
0
votes
1
answer
382
views
How do outliers and missing values impact these classifiers?
I am currently working with a bunch of classification models especially Logistic regression, KNN, Naive Bayes, SVM, and Decision Trees for my machine learning class.
I know how to handle finding and ...
0
votes
1
answer
23
views
Optimization of a simple M x N dataset
I have a dataset consisting of M questionnaires and N students. Each students replied to some questionnaires.
I would like to make the dataset better, by removing some questionnaires and/or some ...
1
vote
1
answer
83
views
What to do when you are building a feature and the denominator is zero?
This is something that looks very simple to solve, but I couldn't find any hint - perhaps I'm not asking Google the right question.
Let's say you own an Internet Company. You have the total ...
- 11
0
votes
1
answer
55
views
Handling missing values in medical data
I have a medical dataset that contains maternal and foetal data during pregnancy. There are some missing values in the dataset that I am unsure how to handle.
Here is a short example of my dataset:
<...
- 417
1
vote
1
answer
56
views
How to fill missing values in a discrete column in sales predictions for a drug supply chain company
I have been working on a dataset that has data from a famous drug supply chain company. The first few records of the dataset look like the following;
Another data accompanies this (primary) dataset. ...
0
votes
1
answer
64
views
Outlier detection - missing values
I have a data science challenge in which two datasets are provided, the first one contains weather data (temperature, wind speed, and precipitation) for a number of days, and the other contains flight ...
- 1
1
vote
0
answers
17
views
Comparing two models with different (naive) baseline
I would like to compare a model with listwise deletion to a model with multiple imputation. However, the model with listwise deletion has a majority class of 70%, while the model with multiple ...
- 11
1
vote
2
answers
78
views
Handling missing data for the majority class
I'm working with an unbalanced (10:1) dataset for classification. I also have a bunch of missing data on certain columns. If I discard them all, I still have a 5:1 ratio, so I guess I can afford to ...
1
vote
0
answers
231
views
How does Kalman filtering work backwards?
For time series missing value imputation I am using the Kalman filter/smoothing approach, given in the imputeTS package.
As Kalman filter is iterative and needs a view data points to make its estimate ...
- 145