Questions tagged [missing-data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
26 views

What is the difference between “fit_transform” and “transform” methods when using “SimpleImputer”? [duplicate]

I have following code, I am not able to understand the difference between use of fit_transform() and transform() method in this ...
8
votes
4answers
321 views

How to impute Missing values not the usual way?

I have a dataset of 4712 records working on binary classification. Label 1 is 33% and Label 0 is 67%. I can't drop records because my sample is already small. Because there are few columns which has ...
1
vote
1answer
14 views

Replacing missing values with mean of feature calculated from previously replaced values

I am not sure how to ask this, but I will try my best. I have replaced some missing values in a feature with the mean of the feature conditional on a second categorical feature. However, not all ...
3
votes
1answer
39 views

How the algorithms such as Xgboost and LightGBM fills the NAN (missing data)?

How the algorithms such as Xgboost and LightGBM fills the NAN ? I know that they learn how to fill it but how they do it is my question and what is they method of learning how to fill the NAN?
2
votes
1answer
21 views

Making predictions with missing numeric data

I don't know much about how to deal with missing data. When the data is categorical it doesn't seem too bad, if I one-hot encode it without dropping any of the actual categories, the missing data is ...
2
votes
0answers
17 views

Distinction of different types of missing values is lost after importing data from SPSS into R

I've got a file containing survey data in SPSS. There are 3 types of missing values defined: invalid (coded as 900), not applicable (990), not filled in (999). After importing the SPSS file into R ...
0
votes
0answers
28 views

Linear regression plotting abline()

I have a ufc data set which i got and started cleaning for my own practice This the link to data:-https://www.kaggle.com/rajeevw/ufcdata#raw_fighter_details.csv and using ...
2
votes
1answer
51 views

Could I add a one hot encoding to each feature representing “has data” versus “has no data”

I have a data set that has some holes in it. I was wondering if I could add two columns for each feature representing this feature has data and ...
0
votes
0answers
19 views

Unseing label in PySpark MultilayerPerceptronClassifier

I'm trying to perform classification with a MLP in PySpark: ...
0
votes
1answer
29 views

Fill the missing values (NA) in various columns (independently of each other) using imputeTS package (in particular, na_kalman function)

A friend of mine has recently started working on R-studio and is interested in filling the NA values in different columns using the above-mentioned function. Also, since he intends to run a time ...
2
votes
1answer
44 views

How to implement single Imputation from conditional distribution?

In [*] page 264, a method of drawing a missing value from a conditional distribution $P(\bf{x}_{mis}|\bf{x}_{obs};\theta)$ which is defined as: I did not find any code implementation of this ...
0
votes
2answers
26 views

How to Keep Missing Values in Ordinal Logistic Regression

I’m using mord package in python to do ordinal logit regression (predict response to movie rating 1-5 stars). One of my predictor variables is also ordinal but ...
0
votes
0answers
21 views

How to handle missing data in a logistic regression?

I am building a model to solve a binary classification task. So far, the input is low dimensional (10 dimensions at most). I need to face the occurrences of missing input. It is my first time at ...
0
votes
0answers
72 views

Suggestion for handling specific missing data

I have data, that describes distance from given location to nearest object (e.g. school, shop etc). Because of performance reasons I couldn't scrape the data about objects, that are futher away than 2....
0
votes
1answer
51 views

How to print nullity correlation matrix

I've a trainingset which has 400 features and most of them have null value. I tried to draw the heatmap of nullity correlation matrix by means of Python and ...
0
votes
1answer
45 views

Replace missing values of numerical features with unique numbers

Let's suppose that I have a dataset with 5 numerical features of which each of them has some missing values and all of them have only non negative values. Some suggested ways to deal with missing ...
0
votes
0answers
28 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
1
vote
1answer
36 views

Encode missing data and unseen data

Let's assume that I have a classification problem and all my features are categorical data. I have missing data (and I do not want to do any imputation). Also, I know that I will have some unseen ...
0
votes
0answers
26 views

Linear/Logistic Regression for unknown values or how to get a good prior for new coefficients

Suppose, we model the probability of making holidays by country and town. The input data are people and how many people actually made holiday in that particular town: ...
0
votes
3answers
55 views

Dealing with informative missingness

How can I deal with a time series that contains missing data which means something? So the value that is missing is not wrong. It's missing on purpose and imputing those missing values would mean a ...
2
votes
3answers
64 views

Dealing with no data

I am working on predictive maintenance and get temperature data from assets. In few months or few days asset remains down and we do not get temperature value. In this scenario i cannot fill data with ...
0
votes
0answers
21 views

Fill na for categorical variables [duplicate]

I am working on a dataset with around 250 rows. There are 2 columns in the dataset gender and education which are generally ...
0
votes
0answers
16 views

What is the default method for Orange to deal with missing values in random forest classification?

Good day, I have built a random forest classification model in Orange, but some of my input data (all continuous) are missing. Up to 30% of the data for some variables are missing. I understand the ...
6
votes
2answers
42 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
0
votes
1answer
26 views

Handling unwanted negative numbers

With me is a dataset collected from IoT sensors with one column labeled “Soil Humidity” measured in percentage. It stands to reason then that all the values be positive percentages, however there’s a ...
0
votes
1answer
30 views

What is the difference between the value -99 and NaN in a data column?

I am new to data science. I was looking into some datasets and I saw some values like -99, which I discovered later that it means that there is a missing value. Does this mean the same thing as NaN? ...
0
votes
0answers
9 views

What's the meaning of Err in Online Time Series Predicion with Missing Values

I'm reading THIS paper about online predictions on time series with missing values. And trying to code the third algorithm in C++. The thing is that I don’t understand what they mean by $Err_{\tau}...
0
votes
0answers
9 views

How to combine multiple data sources where subjects are unknown and there is no ID attribute to match subjects?

I have two data sets. Data set A: data about lifestyles, health conditions and SOCIOECONOMIC and Data set B: data about travel behavior and also SOCIOECONOMIC. We don't know whether they are the same ...
0
votes
0answers
13 views

How to handle variables with text and number?

This is a post with two related questions in one. The first question is: What is the correct procedure when I have variables with different kind of information? Imagine you have a column which has ...
0
votes
1answer
34 views

Do word embeddings help with out of vocab tokens?

I am performing sentiment analysis on a custom dataset of text with Keras but am a little confused about word embeddings. I have been able to train an "Embedding" layer and have also learned to load ...
0
votes
0answers
27 views

How to choose feature which is used to fill another feature's missng values?

I am dealing with a house prediction problem. However, it has about 10% missing values in buildYear which is one of the most important features. I tried filling ...
4
votes
3answers
6k views
0
votes
0answers
68 views

Anomaly detection in time series data from multiple sensors

I've build a classification model based on 15 features coming in real time from 15 different sensors. The window time is 60 seconds (which means that the classification model needs 60 records from ...
0
votes
0answers
13 views

“Best guess” identification of previously seen entities with imperfect data

First of all, I hope that this question does not fall too far out of scope. Choice of approach is certainly discussion-heavy, but I don't consider it unanswerable. I am importing data (person records)...
4
votes
3answers
361 views

What predictive model to use to impute Gender?

My data looks like this: birth_date has 634,990 missing values gender has 328,849 missing values Both of these are a ...
0
votes
0answers
17 views

How to use marker values to deal with missing values?

could someone tell me how to use marker values to deal efficiently with missing values for numerical and categorical features ? I am mainly working with pandas and sklearn libraries.
0
votes
1answer
66 views

Time Series - Values recorded every 10 minutes - Fill missing values

I have a time series data from a sensor that records value periodically - sometimes - every 10 minute period, other times every 5 minute period etc. I have to find out anomalies in real time (as and ...
1
vote
0answers
19 views

How to incorporate an attribute that only exists in some observations?

In a binary classification problem, some of my observations have an event that occurs. I can, obviously, add a 1/0 flag if the event occurs ("event_occurred" in the data below). However, my intuition ...
1
vote
2answers
93 views

Column With Many Missing Values (36%)

Hello this is my first machine learning project, I got a dataset with 18.000 rows and I have a column with 4244 values missing. I don't know why the values are missing since when it's appropriate ...
2
votes
1answer
166 views

Data prepration for logistic regression : Value either “not available” or a “year”

I have some data of houses that have been renovated. In my data there is one column (among others) that captures this information. It is either "-1" if there has not been yet any renovation, or the ...
3
votes
1answer
35 views

Cluster Analysis - Comparing Same Individuals Clustered Across Different Datasets with different features

I have an interesting problem, and I think my Google is failing me since I can't find the same problem anywhere. I have a set of individuals. I have 4 different datasets, with (some) to (all) of ...
1
vote
0answers
14 views

Finding features for missing classes

I have some data with labels from n-1 classes. There is no datapoint with the n-th class in the data but we know that data from the n-th exist. How can we generate data from the missing n+1-the class? ...
2
votes
1answer
30 views

Training on data with inherently non-applicable data cells

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense. Chemical Dataset Contains thousands of rows of chemical mixtures with many columns of ...
0
votes
0answers
14 views

Calculating target mean to validate if I should drop column with missing values is correct?

I am working on the KDD 2009 Cup Data Set (The Small one) and I have a question about preprocessing data. It has a lot of columns with null values, some of them have more than 90% of missing. Reading ...
2
votes
1answer
2k views

How to fill in missing value of the mean of the other columns?

I had a movie dataset including 'budget' and 'genres' attributes. I'd like to fill in the missing value of budget with the mean budget of each genre. I first create two dataframes with or without ...
3
votes
2answers
370 views

How does the naive Bayes classifier handle missing data in testing?

Assume that a classier has been trained already (no missing training data), but a prediction has been requested based on an observation that does not include every feature. How can we handle this ...
0
votes
0answers
65 views

Finding unaffected data in a dataset

The data set has missing values. Further examination tells that they are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data (in percentage)...
2
votes
2answers
30 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
1
vote
2answers
63 views

Handling NA Values in the Chicago Crime Rate data set

I am doing a little project on the Chicago Crime Rate data set and I noticed that there are over 600,000 NA values, primarily in the location fields. I feel that ...
1
vote
1answer
58 views

R mice doesn't give a 'valid' sollution

EDITED: See below for additional information.. TL;DR: How can I add missing data in a dataset like the sample in a way that it doesn't deviate much from the original dataset. ORIGINAL: I have a ...