Questions tagged [missing-data]

Missing data is a problem that arises in data science when some data contained in rows or columns may be missing or unavailable for some samples in a dataset.

Filter by
Sorted by
Tagged with
0 votes
0 answers
36 views

Missing values handling in LightGBM

I'm a bit confused about the handling of missing data by LightGBM. I'm using the R package but my question should not be language-specific. In a regression setting with no categorical feature, I have ...
Augustin's user avatar
  • 101
0 votes
0 answers
7 views

How to selectively train a deep model based on the unavailability of a subset of the feature set

I am creating a deep learning binary classification model. Each sample in the dataset contains two mutually exclusive feature sets X and Y. Feature set X is present in all samples; however, there are ...
flamingo_stark's user avatar
0 votes
1 answer
17 views

Interpolating missing data given county and state totals

Problem: I have population data for states and their constituent counties over several years. Each row is uniquely identified by a state/county and a year. There are four population columns: ...
myusername's user avatar
0 votes
0 answers
41 views

Understanding the working of 'Interpolation'

'Interpolation' is a technique that is used to predict a null/empty value by studying its neighbouring points. Interpolation doesn't take into consideration the entire dataset when predicting the ...
Apoorva's user avatar
  • 275
0 votes
1 answer
57 views

Dropping dataframe columns with missing values

When I looked up different sources that talk about dropping the columns of a dataframe that contains missing values, I got answers that are starkly different from one another. Some sources say, ...
Apoorva's user avatar
  • 275
0 votes
2 answers
20 views

Correlation Analysis dealing with missing value, when the missing values are actually expected

So I am working on the correlation analysis in a dataset and trying to figure out the most sensible way to handle missing values. In my case, the missing values are expected. To make it clearer, here ...
Sijan Bhattarai's user avatar
0 votes
0 answers
75 views

How to input or estimate missing text data?

My task requires 32 different columns with 25 beeing independent text data. Deleting nan values, or cutting columns which has less then 20% (of non NaNs) results in reducing dataset to less then 10% ...
Paweł B's user avatar
0 votes
2 answers
134 views

What is the state-of-the-art in prediction\classification missing labels in partially labeled data?

Overview Let's say I have the following data: ...
Mario's user avatar
  • 364
0 votes
0 answers
31 views

Handling Missing Values

I have the following artificial data : Now I am trying to handle missing values in the age and salary columns using mean imputation. I am using the following code to do so : ...
John adams's user avatar
0 votes
1 answer
72 views

Is it essential to remove NULL values when doing EDA?

I'm doing a Data Analytics project on Youtube Statistics. Assume that I want to get into the entertainment industry and decided to make a youtube channel. Before that, I need to identify what content ...
Leo Tang's user avatar
0 votes
0 answers
22 views

Interpolation techniques predicting negative values in a column with positive values

I tried different interpolation techniques & for some numeric columns with only positive values the interpolation techniques: Polynomial, Cubic Spline & Radial Basis are predicting negative ...
Apoorva's user avatar
  • 275
0 votes
1 answer
31 views

Best strategy for handling missing groups of features

I am currently working on a ML problem where the features used for modelling are sourced from different places/providers. It is very unlikely to find the features from all the different sources to be ...
recentadvances's user avatar
0 votes
1 answer
84 views

How to handle date columns in a dataset with over 100+ date columns?

Background: I have a dataset with around 85k rows and 320 columns.I have no formal domain knowledge and the columns ain't intuitive as well the dataset is not in a language that I speak or understand. ...
Krish Athreyam's user avatar
0 votes
1 answer
105 views

Feature selection / missing values

What are the top (including new, if any) algorithms to perform features selections without removing or altering the missing data points ? Thanks
Zak's user avatar
  • 1
0 votes
1 answer
94 views

Splitting CSV data with missing target variable using train_test_split in Python

I have a dataset in CSV format that consists of a training dataset with around 300 instances and a test dataset with around 100 instances. The issue is that the target variable (the column we want to ...
Yasmine Bzgn's user avatar
0 votes
1 answer
49 views

How to treat missing values depending on what missing means

I have a dataset with quotes from an insurance company. I am trying to create a model to predict how much should the company charge the customer according to the different variables. Two of the ...
Javi's user avatar
  • 103
0 votes
1 answer
140 views

What's the best approach to dealing with missing data in a dataset?

I have a dataset that contains missing values in some columns. I would like to know what is the best approach to deal with this missing data. Should I remove rows with missing data or fill in missing ...
Horbeam's user avatar
0 votes
2 answers
30 views

Data imputation for heavily missing features

I am currently working on the dataset IEEE-CIS Fraud Detection, provided via Kaggle, with around 350 features, with around 600k instances. However, some features are missing large amounts of values, ...
Hai Nguyen's user avatar
0 votes
1 answer
34 views

important feature selection using dimensionality reduction algorithms

I have a dataset having more than 25000 features. I did perform noise removal using the histogram approach, and this dataset gets reduced to more than 5000 features. There are two classes, healthy and ...
Gajanan Kothawade's user avatar
0 votes
1 answer
23 views

Best practice for variables that only have answer if yes in previous column

I currently have a dataset that consists of survey data that has several columns that have answers dependent on the previous question. For example, I may have a question that says "Did you take ...
RLB's user avatar
  • 23
7 votes
5 answers
530 views

How to handle missing value if imputation doesnt make sense

I have column/feature in my dataset showing years a person has been married "years_married". Since not every person is married there are NaN fields. It does not make sense to fillna(0) "...
methus's user avatar
  • 111
4 votes
3 answers
281 views

Imputation of missing values based on target variable

I want to impute missing values in German Credit Risk dataset. ...
Ars ML's user avatar
  • 61
2 votes
1 answer
193 views

Surrogate splits in Python

I want to use RandomForestClassifier from Sklearn to predict categorical variable (credit risk). But one of the predictors seems to have missing values: ...
Ars ML's user avatar
  • 61
0 votes
0 answers
41 views

Imputing Survival Analysis Data in R

The dataset contains 418 observations and 18 total variables. The two dependent variables are N_Days and Status. The dependent ...
E_J7's user avatar
  • 3
0 votes
0 answers
10 views

How to approach a problem of identifying misclassified categories

So I have a dataset of where people have to classify repair cases narratives according to a dropdown. The dropdown has a default type and my boss informs me that an unknown number are probably ...
Wesley Young's user avatar
0 votes
0 answers
41 views

What are the advantages of keeping NAN values instead of handling/replacing them? Pandas

In Pandas, what are the benefits of keeping the nan values in the DataFrame/Series instead of replacing/handling them with another value? Can we use them to indicate an explicit state or flag? Are ...
Chong Onn Keat's user avatar
0 votes
0 answers
14 views

Aggregating irregular recorded accelerometer and gps data

I have a dataset containing accelerometer- and GPS data on which I want to try some classification. However, I am not sure how to aggregate this data into a larger timescale (per day or per week) as ...
Jack_conjo's user avatar
0 votes
1 answer
28 views

How to generate missing values for categorical attributes on a dataset?

I am working for my thesis on 3 known datasets adult,titanic and compass receidivism and i am trying to generate missing values for different missing rates on attributes(e.g Sex,Race) that contain ...
Filippos_p's user avatar
0 votes
1 answer
920 views

Trying to fill null values with sub-grouped mean value using pandas fillna() and groupby().transform() is doing nothing with the null values

I'm doing EDA on the US police shooting dataset where I'm stuck with filling the 482 null values for the age column. Since no strategy was mentioned on how to fill ...
forest's user avatar
  • 111
0 votes
1 answer
155 views

How do I fill the null ages of a person according to the average age of males or females in a particular race

I got this US police shootings dataset where many age are null. Each person's race and gender...
forest's user avatar
  • 111
1 vote
1 answer
31 views

Imputing missing values for "days ago" feature

I have a dataset with features such as last_visit_n_days_ago, last_purchase_n_days_ago. These features are unavailable for many ...
B Arkadievich's user avatar
0 votes
1 answer
97 views

Handling NaN values in a feature representing prioritization ranking

I have a data set like so ...
bananaboy's user avatar
  • 131
0 votes
0 answers
10 views

Understanding model behavior

I am checking the accuracy of a new model (Kmeans with Iterative RF) over SICE (Single Imputation Of Chained Equation) for missing data imputation. For 5-10% SICE is performing well thereafter our ...
Vishnu's user avatar
  • 1
0 votes
1 answer
43 views

Variable selection and NA

I have a very large dataset with a lot of NAs in the data. I want to perform an analysis and have to select the variables that are of most interest. I feel like I have to take 3 steps before I can ...
Wilko's user avatar
  • 13
0 votes
1 answer
53 views

Missing Standard Deviation Values

Currently trying to conduct a meta-analysis. I have reviewed the references I would like to use and made sure each reference met my criteria for this meta-analysis. However, majority of the references ...
Grace's user avatar
  • 1
1 vote
1 answer
60 views

Which is the right imputation method for stock data?

I am currently working on a project involving stock close prices of some companies. I encounter the problem of missing data in the dataset. For example in the image below, because, I have some null ...
Jay Nguyen's user avatar
1 vote
1 answer
33 views

Continuous variables with missing values of type MNAR

I have continuous variables (weather features) with missing values of type MNAR (a different distribution with and without missing values). I learned that these variables should be transformed into ...
Sigal Cohen's user avatar
0 votes
2 answers
234 views

Missing Values Correlation

Is it worth it to study missingness correlation between columns? If you have strongly correlated missing values (say between two columns, A and B), how will this change or shape the way you look at ...
dbzadnen khiari's user avatar
1 vote
0 answers
91 views

Error from XGBoost missing data handling

I have a regression problem with a very large dataset >50 million rows, 81 features and 1 target, all positive float values unevenly distributed between 0 - 1 million. I've trained an XGBoost model ...
lexan55's user avatar
  • 36
0 votes
1 answer
54 views

How to compare between two methods of multivariate to filling NA

In the Titanic dataset, I performed two methods to fill Age NA. The first one is regression using Lasso: ...
Husam Khiry's user avatar
0 votes
2 answers
219 views

Correlations with NA or with Zeros?

When calculating correlations in R e.g. via cor is it better to treat missing data as NAs or as ...
Ben's user avatar
  • 540
1 vote
1 answer
109 views

How to deal with missing values that are supposed to be missing?

I am trying to predict loan defaults with a fairly moderate-sized dataset. I will probably be using logistic regression and random forest. I have around 35 variables and one of them classifies the ...
IcarusX's user avatar
  • 13
0 votes
1 answer
380 views

How do outliers and missing values impact these classifiers?

I am currently working with a bunch of classification models especially Logistic regression, KNN, Naive Bayes, SVM, and Decision Trees for my machine learning class. I know how to handle finding and ...
Vishnu dut's user avatar
0 votes
1 answer
23 views

Optimization of a simple M x N dataset

I have a dataset consisting of M questionnaires and N students. Each students replied to some questionnaires. I would like to make the dataset better, by removing some questionnaires and/or some ...
Samuel-Zacharie Faure's user avatar
1 vote
1 answer
83 views

What to do when you are building a feature and the denominator is zero?

This is something that looks very simple to solve, but I couldn't find any hint - perhaps I'm not asking Google the right question. Let's say you own an Internet Company. You have the total ...
asdlevel1guy's user avatar
0 votes
1 answer
55 views

Handling missing values in medical data

I have a medical dataset that contains maternal and foetal data during pregnancy. There are some missing values in the dataset that I am unsure how to handle. Here is a short example of my dataset: <...
sums22's user avatar
  • 417
1 vote
1 answer
55 views

How to fill missing values in a discrete column in sales predictions for a drug supply chain company

I have been working on a dataset that has data from a famous drug supply chain company. The first few records of the dataset look like the following; Another data accompanies this (primary) dataset. ...
Ritik P. Nayak's user avatar
0 votes
1 answer
64 views

Outlier detection - missing values

I have a data science challenge in which two datasets are provided, the first one contains weather data (temperature, wind speed, and precipitation) for a number of days, and the other contains flight ...
Alex's user avatar
  • 1
1 vote
0 answers
16 views

Comparing two models with different (naive) baseline

I would like to compare a model with listwise deletion to a model with multiple imputation. However, the model with listwise deletion has a majority class of 70%, while the model with multiple ...
Tessa's user avatar
  • 11
1 vote
2 answers
78 views

Handling missing data for the majority class

I'm working with an unbalanced (10:1) dataset for classification. I also have a bunch of missing data on certain columns. If I discard them all, I still have a 5:1 ratio, so I guess I can afford to ...
the_underfitter's user avatar

1
2 3 4 5