Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [missing-data]

Missing data is a problem that arises in data science when some data contained in rows or columns may be missing or unavailable for some samples in a dataset.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Missing Value in a dataset

i'm curently cleaning a dataset and theres a column called " Institution", it is needed for encoding and training the classification model later so it needs to be cleaned. In that column, ...
Harry lou's user avatar
2 votes
0 answers
18 views

Best practices for handling "NA" when all NA values exist due to being below the limit of detection?

I am working in R, and have a data set which has a few metabolite concentration values as continuous variables. Anywhere that the concentration was too low to be detected it simply says <LOD. This ...
KLN-RDN's user avatar
  • 21
1 vote
0 answers
51 views

Imputing techniques for missing values

Currently, I am working on a project for which I would appreciate some feedback and opinions. My dataset contains data about daily solar radiation from 8 stations and covers the period from 2017 to ...
Fatima's user avatar
  • 111
0 votes
1 answer
48 views

How to deal with missing values

I know this topic has been extensively covered, but I haven't found an answer that suits my needs. I'm currently interning and working on electronic boards. These electronic boards go through test ...
naomit's user avatar
  • 1
3 votes
0 answers
112 views

Why is multiple imputation not used more widely in Data Science?

I have a background in statistics. Multiple imputation is very commonly used to handle missing data, and if it is not used it almost always results in serious criticism. Recently I have been ...
Joe King's user avatar
  • 141
0 votes
0 answers
7 views

How to handle inherent missing data in constructed lagging indicators?

I have a dataset of sports statistics from over the course of several seasons, and I want to incorporate some lagging features for a deep learning time series model. Specifically, I want to generate ...
moistnar's user avatar
0 votes
0 answers
145 views

Missing values handling in LightGBM

I'm a bit confused about the handling of missing data by LightGBM. I'm using the R package but my question should not be language-specific. In a regression setting with no categorical feature, I have ...
Augustin's user avatar
  • 101
0 votes
0 answers
10 views

How to selectively train a deep model based on the unavailability of a subset of the feature set

I am creating a deep learning binary classification model. Each sample in the dataset contains two mutually exclusive feature sets X and Y. Feature set X is present in all samples; however, there are ...
flamingo_stark's user avatar
0 votes
1 answer
24 views

Interpolating missing data given county and state totals

Problem: I have population data for states and their constituent counties over several years. Each row is uniquely identified by a state/county and a year. There are four population columns: ...
myusername's user avatar
0 votes
0 answers
50 views

Understanding the working of 'Interpolation'

'Interpolation' is a technique that is used to predict a null/empty value by studying its neighbouring points. Interpolation doesn't take into consideration the entire dataset when predicting the ...
Apoorva's user avatar
  • 307
0 votes
1 answer
1k views

Dropping dataframe columns with missing values

When I looked up different sources that talk about dropping the columns of a dataframe that contains missing values, I got answers that are starkly different from one another. Some sources say, ...
Apoorva's user avatar
  • 307
0 votes
2 answers
63 views

Correlation Analysis dealing with missing value, when the missing values are actually expected

So I am working on the correlation analysis in a dataset and trying to figure out the most sensible way to handle missing values. In my case, the missing values are expected. To make it clearer, here ...
Sijan Bhattarai's user avatar
0 votes
0 answers
81 views

How to input or estimate missing text data?

My task requires 32 different columns with 25 beeing independent text data. Deleting nan values, or cutting columns which has less then 20% (of non NaNs) results in reducing dataset to less then 10% ...
Paweł B's user avatar
0 votes
2 answers
158 views

What is the state-of-the-art in prediction\classification missing labels in partially labeled data?

Overview Let's say I have the following data: ...
Mario's user avatar
  • 400
0 votes
0 answers
35 views

Handling Missing Values

I have the following artificial data : Now I am trying to handle missing values in the age and salary columns using mean imputation. I am using the following code to do so : ...
John adams's user avatar
0 votes
1 answer
312 views

Is it essential to remove NULL values when doing EDA?

I'm doing a Data Analytics project on Youtube Statistics. Assume that I want to get into the entertainment industry and decided to make a youtube channel. Before that, I need to identify what content ...
Leo Tang's user avatar
0 votes
0 answers
28 views

Interpolation techniques predicting negative values in a column with positive values

I tried different interpolation techniques & for some numeric columns with only positive values the interpolation techniques: Polynomial, Cubic Spline & Radial Basis are predicting negative ...
Apoorva's user avatar
  • 307
0 votes
1 answer
34 views

Best strategy for handling missing groups of features

I am currently working on a ML problem where the features used for modelling are sourced from different places/providers. It is very unlikely to find the features from all the different sources to be ...
recentadvances's user avatar
0 votes
1 answer
394 views

How to handle date columns in a dataset with over 100+ date columns?

Background: I have a dataset with around 85k rows and 320 columns.I have no formal domain knowledge and the columns ain't intuitive as well the dataset is not in a language that I speak or understand. ...
Krish Athreyam's user avatar
0 votes
1 answer
405 views

Feature selection / missing values

What are the top (including new, if any) algorithms to perform features selections without removing or altering the missing data points ? Thanks
Zak's user avatar
  • 1
0 votes
1 answer
253 views

Splitting CSV data with missing target variable using train_test_split in Python

I have a dataset in CSV format that consists of a training dataset with around 300 instances and a test dataset with around 100 instances. The issue is that the target variable (the column we want to ...
Yasmine Bzgn's user avatar
0 votes
1 answer
54 views

How to treat missing values depending on what missing means

I have a dataset with quotes from an insurance company. I am trying to create a model to predict how much should the company charge the customer according to the different variables. Two of the ...
Javi's user avatar
  • 103
0 votes
1 answer
498 views

What's the best approach to dealing with missing data in a dataset?

I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing ...
Horbeam's user avatar
0 votes
2 answers
35 views

Data imputation for heavily missing features

I am currently working on the dataset IEEE-CIS Fraud Detection, provided via Kaggle, with around 350 features, with around 600k instances. However, some features are missing large amounts of values, ...
Hai Nguyen's user avatar
0 votes
1 answer
42 views

important feature selection using dimensionality reduction algorithms

I have a dataset having more than 25000 features. I did perform noise removal using the histogram approach, and this dataset gets reduced to more than 5000 features. There are two classes, healthy and ...
Gajanan Kothawade's user avatar
0 votes
1 answer
26 views

Best practice for variables that only have answer if yes in previous column

I currently have a dataset that consists of survey data that has several columns that have answers dependent on the previous question. For example, I may have a question that says "Did you take ...
RLB's user avatar
  • 23
7 votes
5 answers
687 views

How to handle missing value if imputation doesnt make sense

I have column/feature in my dataset showing years a person has been married "years_married". Since not every person is married there are NaN fields. It does not make sense to fillna(0) "...
methus's user avatar
  • 111
4 votes
4 answers
420 views

Imputation of missing values based on target variable

I want to impute missing values in German Credit Risk dataset. df['Saving accounts'].value_counts(dropna=False) output: ...
Ars ML's user avatar
  • 61
2 votes
1 answer
267 views

Surrogate splits in Python

I want to use RandomForestClassifier from Sklearn to predict categorical variable (credit risk). But one of the predictors seems to have missing values: ...
Ars ML's user avatar
  • 61
0 votes
1 answer
32 views

How to generate missing values for categorical attributes on a dataset?

I am working for my thesis on 3 known datasets adult,titanic and compass receidivism and i am trying to generate missing values for different missing rates on attributes(e.g Sex,Race) that contain ...
Filippos_p's user avatar
0 votes
1 answer
2k views

Trying to fill null values with sub-grouped mean value using pandas fillna() and groupby().transform() is doing nothing with the null values

I'm doing EDA on the US police shooting dataset where I'm stuck with filling the 482 null values for the age column. Since no strategy was mentioned on how to fill ...
forest's user avatar
  • 111
0 votes
1 answer
254 views

How do I fill the null ages of a person according to the average age of males or females in a particular race

I got this US police shootings dataset where many age are null. Each person's race and gender...
forest's user avatar
  • 111
1 vote
1 answer
36 views

Imputing missing values for "days ago" feature

I have a dataset with features such as last_visit_n_days_ago, last_purchase_n_days_ago. These features are unavailable for many ...
B Arkadievich's user avatar
0 votes
1 answer
111 views

Handling NaN values in a feature representing prioritization ranking

I have a data set like so ...
bananaboy's user avatar
  • 131
0 votes
1 answer
71 views

Variable selection and NA

I have a very large dataset with a lot of NAs in the data. I want to perform an analysis and have to select the variables that are of most interest. I feel like I have to take 3 steps before I can ...
Wilko's user avatar
  • 13
0 votes
1 answer
75 views

Missing Standard Deviation Values

Currently trying to conduct a meta-analysis. I have reviewed the references I would like to use and made sure each reference met my criteria for this meta-analysis. However, majority of the references ...
Grace's user avatar
  • 1
1 vote
1 answer
73 views

Which is the right imputation method for stock data?

I am currently working on a project involving stock close prices of some companies. I encounter the problem of missing data in the dataset. For example in the image below, because, I have some null ...
Jay Nguyen's user avatar
1 vote
1 answer
35 views

Continuous variables with missing values of type MNAR

I have continuous variables (weather features) with missing values of type MNAR (a different distribution with and without missing values). I learned that these variables should be transformed into ...
Sigal Cohen's user avatar
0 votes
2 answers
332 views

Missing Values Correlation

Is it worth it to study missingness correlation between columns? If you have strongly correlated missing values (say between two columns, A and B), how will this change or shape the way you look at ...
dbzadnen khiari's user avatar
1 vote
0 answers
107 views

Error from XGBoost missing data handling

I have a regression problem with a very large dataset >50 million rows, 81 features and 1 target, all positive float values unevenly distributed between 0 - 1 million. I've trained an XGBoost model ...
lexan55's user avatar
  • 36
0 votes
1 answer
58 views

How to compare between two methods of multivariate to filling NA

In the Titanic dataset, I performed two methods to fill Age NA. The first one is regression using Lasso: ...
Husam Khiry's user avatar
0 votes
2 answers
286 views

Correlations with NA or with Zeros?

When calculating correlations in R e.g. via cor is it better to treat missing data as NAs or as ...
Ben's user avatar
  • 560
1 vote
1 answer
123 views

How to deal with missing values that are supposed to be missing?

I am trying to predict loan defaults with a fairly moderate-sized dataset. I will probably be using logistic regression and random forest. I have around 35 variables and one of them classifies the ...
IcarusX's user avatar
  • 13
0 votes
1 answer
571 views

How do outliers and missing values impact these classifiers?

I am currently working with a bunch of classification models especially Logistic regression, KNN, Naive Bayes, SVM, and Decision Trees for my machine learning class. I know how to handle finding and ...
Vishnu dut's user avatar
0 votes
1 answer
23 views

Optimization of a simple M x N dataset

I have a dataset consisting of M questionnaires and N students. Each students replied to some questionnaires. I would like to make the dataset better, by removing some questionnaires and/or some ...
Samuel-Zacharie Faure's user avatar
1 vote
1 answer
114 views

What to do when you are building a feature and the denominator is zero?

This is something that looks very simple to solve, but I couldn't find any hint - perhaps I'm not asking Google the right question. Let's say you own an Internet Company. You have the total ...
asdlevel1guy's user avatar
0 votes
1 answer
63 views

Handling missing values in medical data

I have a medical dataset that contains maternal and foetal data during pregnancy. There are some missing values in the dataset that I am unsure how to handle. Here is a short example of my dataset: <...
sums22's user avatar
  • 427
1 vote
1 answer
65 views

How to fill missing values in a discrete column in sales predictions for a drug supply chain company

I have been working on a dataset that has data from a famous drug supply chain company. The first few records of the dataset look like the following; Another data accompanies this (primary) dataset. ...
Ritik P. Nayak's user avatar
0 votes
1 answer
70 views

Outlier detection - missing values

I have a data science challenge in which two datasets are provided, the first one contains weather data (temperature, wind speed, and precipitation) for a number of days, and the other contains flight ...
Alex's user avatar
  • 1
1 vote
0 answers
18 views

Comparing two models with different (naive) baseline

I would like to compare a model with listwise deletion to a model with multiple imputation. However, the model with listwise deletion has a majority class of 70%, while the model with multiple ...
Tessa's user avatar
  • 11

1
2 3 4 5