Questions tagged [missing-data]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2
votes
0answers
8 views

Distinction of different types of missing values is lost after importing data from SPSS into R

I've got a file containing survey data in SPSS. There are 3 types of missing values defined: invalid (coded as 900), not applicable (990), not filled in (999). After importing the SPSS file into R ...
0
votes
0answers
24 views

Linear regression plotting abline() [[R]]

I have a ufc data set which i got and started cleaning for my own practice This the link to data:-https://www.kaggle.com/rajeevw/ufcdata#raw_fighter_details.csv and using raw_fighter_data file. I ...
2
votes
1answer
40 views

Could I add a one hot encoding to each feature representing “has data” versus “has no data”

I have a data set that has some holes in it. I was wondering if I could add two columns for each feature representing this feature has data and ...
0
votes
0answers
14 views

Unseing label in PySpark MultilayerPerceptronClassifier

I'm trying to perform classification with a MLP in PySpark: ...
0
votes
1answer
25 views

Fill the missing values (NA) in various columns (independently of each other) using imputeTS package (in particular, na_kalman function)

A friend of mine has recently started working on R-studio and is interested in filling the NA values in different columns using the above-mentioned function. Also, since he intends to run a time ...
2
votes
1answer
42 views

How to implement single Imputation from conditional distribution?

In [*] page 264, a method of drawing a missing value from a conditional distribution $P(\bf{x}_{mis}|\bf{x}_{obs};\theta)$ which is defined as: I did not find any code implementation of this ...
0
votes
2answers
23 views

How to Keep Missing Values in Ordinal Logistic Regression

I’m using mord package in python to do ordinal logit regression (predict response to movie rating 1-5 stars). One of my predictor variables is also ordinal but ...
0
votes
0answers
19 views

How to handle missing data in a logistic regression?

I am building a model to solve a binary classification task. So far, the input is low dimensional (10 dimensions at most). I need to face the occurrences of missing input. It is my first time at ...
0
votes
0answers
68 views

Suggestion for handling specific missing data

I have data, that describes distance from given location to nearest object (e.g. school, shop etc). Because of performance reasons I couldn't scrape the data about objects, that are futher away than 2....
0
votes
1answer
33 views

How to print nullity correlation matrix

I've a trainingset which has 400 features and most of them have null value. I tried to draw the heatmap of nullity correlation matrix by means of Python and ...
0
votes
1answer
41 views

Replace missing values of numerical features with unique numbers

Let's suppose that I have a dataset with 5 numerical features of which each of them has some missing values and all of them have only non negative values. Some suggested ways to deal with missing ...
0
votes
0answers
23 views

Filling 2 different values for missing/NaN columns

I am doing a binary classification problem (TARGET = 0 or 1). My dataset contains some NaN ...
2
votes
0answers
19 views

Encode missing data and unseen data

Let's assume that I have a classification problem and all my features are categorical data. I have missing data (and I do not want to do any imputation). Also, I know that I will have some unseen ...
0
votes
0answers
26 views

Linear/Logistic Regression for unknown values or how to get a good prior for new coefficients

Suppose, we model the probability of making holidays by country and town. The input data are people and how many people actually made holiday in that particular town: ...
0
votes
3answers
46 views

Dealing with informative missingness

How can I deal with a time series that contains missing data which means something? So the value that is missing is not wrong. It's missing on purpose and imputing those missing values would mean a ...
2
votes
3answers
63 views

Dealing with no data

I am working on predictive maintenance and get temperature data from assets. In few months or few days asset remains down and we do not get temperature value. In this scenario i cannot fill data with ...
0
votes
0answers
21 views

Fill na for categorical variables [duplicate]

I am working on a dataset with around 250 rows. There are 2 columns in the dataset gender and education which are generally ...
0
votes
0answers
15 views

What is the default method for Orange to deal with missing values in random forest classification?

Good day, I have built a random forest classification model in Orange, but some of my input data (all continuous) are missing. Up to 30% of the data for some variables are missing. I understand the ...
6
votes
2answers
33 views

When to use missing data imputation in the data analysis problem?

I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use ...
0
votes
1answer
26 views

Handling unwanted negative numbers

With me is a dataset collected from IoT sensors with one column labeled “Soil Humidity” measured in percentage. It stands to reason then that all the values be positive percentages, however there’s a ...
0
votes
1answer
28 views

What is the difference between the value -99 and NaN in a data column?

I am new to data science. I was looking into some datasets and I saw some values like -99, which I discovered later that it means that there is a missing value. Does this mean the same thing as NaN? ...
0
votes
0answers
9 views

What's the meaning of Err in Online Time Series Predicion with Missing Values

I'm reading THIS paper about online predictions on time series with missing values. And trying to code the third algorithm in C++. The thing is that I don’t understand what they mean by $Err_{\tau}...
0
votes
0answers
9 views

How to combine multiple data sources where subjects are unknown and there is no ID attribute to match subjects?

I have two data sets. Data set A: data about lifestyles, health conditions and SOCIOECONOMIC and Data set B: data about travel behavior and also SOCIOECONOMIC. We don't know whether they are the same ...
0
votes
0answers
13 views

How to handle variables with text and number?

This is a post with two related questions in one. The first question is: What is the correct procedure when I have variables with different kind of information? Imagine you have a column which has ...
0
votes
1answer
31 views

Do word embeddings help with out of vocab tokens?

I am performing sentiment analysis on a custom dataset of text with Keras but am a little confused about word embeddings. I have been able to train an "Embedding" layer and have also learned to load ...
0
votes
0answers
27 views

How to choose feature which is used to fill another feature's missng values?

I am dealing with a house prediction problem. However, it has about 10% missing values in buildYear which is one of the most important features. I tried filling ...
2
votes
2answers
5k views
0
votes
0answers
63 views

Anomaly detection in time series data from multiple sensors

I've build a classification model based on 15 features coming in real time from 15 different sensors. The window time is 60 seconds (which means that the classification model needs 60 records from ...
0
votes
0answers
13 views

“Best guess” identification of previously seen entities with imperfect data

First of all, I hope that this question does not fall too far out of scope. Choice of approach is certainly discussion-heavy, but I don't consider it unanswerable. I am importing data (person records)...
4
votes
3answers
316 views

What predictive model to use to impute Gender?

My data looks like this: birth_date has 634,990 missing values gender has 328,849 missing values Both of these are a ...
0
votes
0answers
17 views

How to use marker values to deal with missing values?

could someone tell me how to use marker values to deal efficiently with missing values for numerical and categorical features ? I am mainly working with pandas and sklearn libraries.
0
votes
1answer
58 views

Time Series - Values recorded every 10 minutes - Fill missing values

I have a time series data from a sensor that records value periodically - sometimes - every 10 minute period, other times every 5 minute period etc. I have to find out anomalies in real time (as and ...
1
vote
0answers
19 views

How to incorporate an attribute that only exists in some observations?

In a binary classification problem, some of my observations have an event that occurs. I can, obviously, add a 1/0 flag if the event occurs ("event_occurred" in the data below). However, my intuition ...
1
vote
2answers
82 views

Column With Many Missing Values (36%)

Hello this is my first machine learning project, I got a dataset with 18.000 rows and I have a column with 4244 values missing. I don't know why the values are missing since when it's appropriate ...
2
votes
1answer
166 views

Data prepration for logistic regression : Value either “not available” or a “year”

I have some data of houses that have been renovated. In my data there is one column (among others) that captures this information. It is either "-1" if there has not been yet any renovation, or the ...
3
votes
1answer
35 views

Cluster Analysis - Comparing Same Individuals Clustered Across Different Datasets with different features

I have an interesting problem, and I think my Google is failing me since I can't find the same problem anywhere. I have a set of individuals. I have 4 different datasets, with (some) to (all) of ...
1
vote
0answers
13 views

Finding features for missing classes

I have some data with labels from n-1 classes. There is no datapoint with the n-th class in the data but we know that data from the n-th exist. How can we generate data from the missing n+1-the class? ...
2
votes
1answer
28 views

Training on data with inherently non-applicable data cells

I am training a model on a chemical sample dataset to find outliers and perform imputation where it makes sense. Chemical Dataset Contains thousands of rows of chemical mixtures with many columns of ...
0
votes
0answers
13 views

Calculating target mean to validate if I should drop column with missing values is correct?

I am working on the KDD 2009 Cup Data Set (The Small one) and I have a question about preprocessing data. It has a lot of columns with null values, some of them have more than 90% of missing. Reading ...
2
votes
1answer
1k views

How to fill in missing value of the mean of the other columns?

I had a movie dataset including 'budget' and 'genres' attributes. I'd like to fill in the missing value of budget with the mean budget of each genre. I first create two dataframes with or without ...
3
votes
2answers
342 views

How does the naive Bayes classifier handle missing data in testing?

Assume that a classier has been trained already (no missing training data), but a prediction has been requested based on an observation that does not include every feature. How can we handle this ...
0
votes
0answers
64 views

Finding unaffected data in a dataset

The data set has missing values. Further examination tells that they are spread along 1.5 standard deviation from the median with distribution mean = 0 & variance = 5. How much data (in percentage)...
2
votes
2answers
29 views

How to think about - and sometimes impute - geographic distances

I have a dataset with one of the (important) features being the geographic distances from NYC. Of course, some of the values are missing.... The goal is predicting whether people with certain ...
1
vote
1answer
53 views

Handling NA Values in the Chicago Crime Rate data set

I am doing a little project on the Chicago Crime Rate data set and I noticed that there are over 600,000 NA values, primarily in the location fields. I feel that ...
1
vote
1answer
57 views

R mice doesn't give a 'valid' sollution

EDITED: See below for additional information.. TL;DR: How can I add missing data in a dataset like the sample in a way that it doesn't deviate much from the original dataset. ORIGINAL: I have a ...
2
votes
0answers
32 views

Detect Missing Records in Dataset

I have a dataset that contains several measures from various widgets on a daily basis. While the widgets remain relatively stable over time, sometimes there are legitimate reasons for one to disappear ...
1
vote
3answers
61 views

What best/correct algorithm/procedure to cluster a dataset with a lot 0's?

I'm new to statistics so sorry any major lack of knowledge in the topic, just doing a project for graduation. I'm trying to cluster a Health dataset containing Diseases(3456) and Symptoms(25) ...
2
votes
1answer
350 views

How to deal with missing data for Bernoulli Naive Bayes?

I am dealing with a dataset of categorical data that looks like this: ...
3
votes
1answer
432 views

Handling missing values to optimize polynomial features

I was playing around with some data to practice my Python and machine learning skills and wanted to create polynomial features from two features that I think are related and have a strong influence on ...
0
votes
1answer
37 views

Missing Values In New Data

(Before someone marks this as duplicate - I'm not asking about training data, I'm asking about new data which has come in and needs to be classified) Suppose I've got a dataset which has 5 predictors ...