12

Your error is due to using Simple Imputer's fit and fit_transform on a numpy array. Here's how i used it on a Dataframe imr = Imputer(missing_values='NaN', strategy='median', axis=0) imr = imr.fit(data[['age']]) data['age'] = imr.transform(data[['age']]).ravel() X.fit = impute.fit_transform().. this is wrong. you can't assign a value to a X.fit() just ...


9

You can use this function : forcats::fct_explicit_na library(forcats) fct_explicit_na(DF$col, na_level = "None") Usage It can be used within the mutate function and piped to edit DF directly: library(tidyverse) # for tidy data packages, automatically loads dplyr library(magrittr) # for piping DF %<>% mutate(cols = fct_explicit_na(col, na_level =...


9

To decide which strategy is appropriate, it is important to investigate the mechanism that led to the missing values to find out whether the missing data is missing completely at random (MCAR), missing at random (MAR), or missing not at random (MNAR). MCAR means that there is no relationship between the missingness of the data and any of the values. MAR ...


6

You need to add "None" to the factor level and refactor the column DF$col. I added an example script using the iris dataset. df <- iris # set 20 Species to NA set.seed(1234) s <- sample(nrow(df), 20) df$Species[s] <- NA # Get levels and add "None" levels <- levels(df$Species) levels[length(levels) + 1] <- "None" # refactor Species to ...


6

Use the extra features for unsupervised learning. You might enjoy Vladimir Vapnik's take on this in the context of SVMs, which he calls privileged learning: Learning with Intelligent Teacher: Similarity Control and Knowledge Transfer


6

There are three main approaches to handling missing data. Impute - use some method to fill in the missing values with reasonable guesses. You could interpolate between two time points, take the average value over all time points, or use a variety of other techniques leveraging co-occurrence of other variables to get a reasonable estimate. Ignore - some ...


6

A simple approach could be the following: suppose $i \in \{0,1\}^d$ is the vector you want to predict which of the $0$ entries could be $1$ and $j \in J$ the rest of the feature vectors. Take the $k$ nearest neighbors, under some suitable distance (Jaccard, Hamming, Manhattan distance). For each $0$ entry the probabilities could be the percentage of the $k$ ...


6

A trick I have seen on Kaggle. Step 1: replace NAN with the mean or the median. The mean, if the data is normally distributed, otherwise the median. In my case, I have NANs in Age. Step 2: Add a new column "NAN_Age." 1 for NAN, 0 otherwise. If there's a pattern in NAN, you help the algorithm catch it. A nice bonus is that this strategy doesn't care if it'...


5

There are of course other choices to fill in for missing data. The median was already mentioned, and it may work better in certain cases. There may even be much better alternatives, which may be very specific to your problem. To find out whether this is the case, you must find out more about the nature of your missing data. When you understand in detail why ...


5

I don't understand why you would like to fill values with zeros ! This would basically mean, "this guy, who is 170 cm tall, weights 0 kg" and would fool your network. In my opinion, you have two options: discard missing values (the entire row): you end up with less but more consistent training data if you really need these rows, then fill missing values ...


5

You can use one of the popular gradient boosting tree implementations such LightGBM and XGBoost as your predictive model. They can handle missing values during training and the results are often better than any imputation done in preprocessing. The way they achieve this is by using the splitting of the decision trees they are built on. The missing values ...


5

I agree with Simon's advice. I find that the gains that you obtain from using any external method of imputation is often inferior to an internal method, and on top of this, exposes you to even more potential of severely screwing up with respect to data leakage. That being said, besides using an algorithm that automatically handles missing data for you (...


5

Generally speaking you have two options: impute the missing data discard the missing data Due to the fact that ML models perform better with more data, the former is usually preferred. However, you should keep in mind that the imputed data should not affect the distribution of the feature. This is especially the case with features with a high percentage ...


4

We would need more information on the prediction problem and the features to be able to give something more precise. Anyhow, I am surprised no answer so far included all possible options since they aren't that many: get rid of incomplete observations or features --- obviously, only viable if there are few incomplete cases since you lose too much ...


4

When it comes to missing data, there are many different methods of filling these values. However, the imputation method you choose, depends largely on the amount of missing data and the type of variable. For example, you won't impute the mean value for missing categorical data, you would choose the mode instead. No matter which method you choose, there will ...


4

Your question is sensible. The way in which posterior probability is calculated in the classical Naive Bayes classifier (in sklearn) is like summation of the conditional probabilities of the all the features in the dataset. Even though the features are treated as conditionally independent, to learn the classification probability all the features are always ...


4

Since we are talking about multiple different types of targets (classes versus numerical for example) we already need a composite loss function. I will consider how to balance the different composite parts of the loss function outside of the scope of this answer but if you look into multi-task learning there are solutions to this. What you could do (both in ...


4

First use a binary 0 (no renovation) and 1 (renovation) which works perfect with logistic regression. Using the exact date is a bad practice. It guides the model in the direction of over-fitting on specific dates. For example, a pattern from 2006 would be specific to that year and would not help the future years. As an alternative, binning on larger spans ...


4

SimpleImputer also works fine. from sklearn.impute import SimpleImputer imputer=SimpleImputer(missing_values=np.nan,strategy='mean') imputer=imputer.fit(X[:,1:3]) X[:,1:3]=imputer.transform(X[:,1:3]) which gives result array([['France', 44.0, 72000.0], ['Spain', 27.0, 48000.0], ['Germany', 30.0, 54000.0], ['Spain', 38.0, 61000.0], ...


4

LIGHTGBM will ignore missing values during a split, then allocate them to whichever side reduces the loss the most. Section 3.2 of this reference explains it. There are some options you can set such as usemissing=false, which disables handling for missing values. You can also use the zeroas_missing option to change behavior. GitHub


3

Imputation and dealing with missing data a broad subject; you should start by researching standard material on this subject. The first question to figure out is Why is some data missing? and What is the process that causes data to be missing? It's important to understand how this happens, because this will affect what solution is appropriate. Randomly ...


3

I would not definitely recommend substituting missing values by mean or by median or mode. If you want to go through some techniques and get a glance at them, I would recommend going through this link and for imputation techniques this wiki page gives you a brief . Do you think that there is a way to predict missing values from the other cells. If yes, apply ...


3

As told by @smci, this technique is called Data Imputation. There are several techniques which can be used to deal with the missing data. Some of these are: Mean/ Mode/ Median Imputation: Imputation is a method to fill in the missing values with estimated ones. The objective is to employ known relationships that can be identified in the valid values of the ...


3

I think the question you're wrestling with is essentially this: Is there a way to use information that may be present in the data without A as part of your strategy for predicting A? There is actually a name for the set of methods that do exactly that: semi-supervised learning. While there are multiple techniques, a method analogous to what you suggest ...


3

Interpolation seems like it would make sense in this case: any time you miss a day, take an average of the before and after. As an aside, I don't think you have to give up on the missing weather values so easily. There are a variety of R packages that simplify getting weather for an arbitrary location from someone like Weather Underground with only a couple ...


3

Yes there are so many approaches to handle missing data or missing values depending on the task and the property of the data itself. For example in time series you can think about forward filling or even backward filling, max, mean or median over a time lapse. There are also 'smarter' ways like training a model over the available data and try to predict the ...


3

Various methods are available for fill missing values in data. Ignore the tuple is the simplest and not effective method. Fill the missing value manually. Use a global constant to fill the missing value. Use attribute mean value to fill missing value. Use attribute mean for all samples belonging to the same class as the given tuple. Use most probable value ...


3

First of all, if most of your data is missing, you are in trouble anyway. You need to ask why is most of the data missing, and also, why are the data you observed not missing. Being missing is very likely telling you something in your data. All methods of correcting missing data, including the naive interpolation, mean replacement, and median replacement ...


3

Your intuition about 'no effect' is true in some sense. But this replacement may be not the best use of the information you have. The choice of missing value treatment depends on your initial problem statement. In all the cases I assume that you have already somehow estimated conditional means $\mu_0$ and $\mu_1$ and the common variance matrix $S$. ...


3

There are several different approaches here. One (which you've already described) could be to impute missing values with their mean. You could then add an extra column which keeps track of whether that value was originally missing or not. So, in the example you provide, we would end up with id age sex dropout s1_q1 s1_q2 s1_q3 s1_q4 s1_q5.... s5_q10 ...


Only top voted, non community-wiki answers of a minimum length are eligible