Skip to main content
18 votes
Accepted

Which comes first? Multiple Imputation, Splitting into train/test, or Standardization/Normalization

Always split before you do any data pre-processing. Performing pre-processing before splitting will mean that information from your test set will be present during training, causing a data leak. ...
Simon Larsson's user avatar
13 votes
Accepted

Please review my sketch of the Machine Learning process

This process will result in data leaks. The split needs to happen earlier. Normalizing data before the split means that your training data contains information about your test data. I would put the ...
Simon Larsson's user avatar
9 votes

Retrieve dropped column names from `sklearn.impute.SimpleImputer`

I've got the same issue today, and it's a shame your post got no answers. I think this question is not well addressed in the sklearn documentation. I can show you my workaround to this issue: ...
Vicent Blanes's user avatar
7 votes

How to deal with missing data for only some categories

There are three types of missing data: Missing Completely At Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR). Your case is the second, where according to wikipedia it: ...
Mnng's user avatar
  • 311
6 votes

What predictive model to use to impute Gender?

You can use one of the popular gradient boosting tree implementations such LightGBM and XGBoost as your predictive model. They can handle missing values during training and the results are often ...
Simon Larsson's user avatar
6 votes

How to handle missing value if imputation doesnt make sense

I think this is a good solution. You could also try to set a unique negative value for non-married people, especially if you are using a tree-based model.
Just trying's user avatar
6 votes

How to handle missing value if imputation doesnt make sense

You could consider setting years_married to -1, then it is different from columns for the ones that are just married and could thus be understood by a decision tree....
justinlk's user avatar
  • 136
5 votes
Accepted

What predictive model to use to impute Gender?

I agree with Simon's advice. I find that the gains that you obtain from using any external method of imputation is often inferior to an internal method, and on top of this, exposes you to even more ...
aranglol's user avatar
  • 2,196
5 votes

When to use missing data imputation in the data analysis problem?

Generally speaking you have two options: impute the missing data discard the missing data Due to the fact that ML models perform better with more data, the former is usually preferred. However, you ...
ILM91's user avatar
  • 338
4 votes
Accepted

Should I Impute target values?

For the missing data problem, one thing to be aware of, is the missingness mechanism. Depending of the dataset, the NA's (Missing Values) you have could be a result of a condition of the phenomenon ...
az_br's user avatar
  • 56
4 votes
Accepted

Retrieve dropped column names from `sklearn.impute.SimpleImputer`

Since the original question, scikit-learn (version 1.1, May 2022) has implemented get_feature_names_out methods for most (if not all) transformers. Now, column ...
njp's user avatar
  • 181
4 votes
Accepted

Should I deal with missing values first then transform the data or vice versa?

In general, it is better to deal with missing values first because there could be data loss or additional noise applying operations like a log that could impact classification or prediction algorithms....
Nicolas Martin's user avatar
4 votes

Imputing time data for an event that hasn't occurred yet

As so often, the answer is: "It depends". In this case, it depends on the algorithm / model / method you are going to use. There are some methods, e.g. tree-based methods, that can handle ...
Broele's user avatar
  • 1,525
3 votes

Methodologies for predicting missing data

As told by @smci, this technique is called Data Imputation. There are several techniques which can be used to deal with the missing data. Some of these are: Mean/ Mode/ Median Imputation: Imputation ...
enterML's user avatar
  • 3,041
3 votes

What predictive model to use to impute Gender?

If you are planning to go with imputing values as a pre-processing step, I just want to add that it is better to impute missing values of multiple variables jointly because you will be able to better ...
AlexK's user avatar
  • 350
3 votes

Which comes first? Multiple Imputation, Splitting into train/test, or Standardization/Normalization

If you impute/standardize before splitting and then split into train/test you are leaking data from your test set (that is supposed to be completely withheld) into your training set. This will yield ...
aranglol's user avatar
  • 2,196
3 votes

What do I do when my column has 50% data missing?

It depends on your knowledge of the problem. First, you should classify why is it missing?? Structurally missing data Structurally missing data is data that is missing for a logical reason. In other ...
Carlos Mougan's user avatar
3 votes

Please review my sketch of the Machine Learning process

Yes, these are the basics step. Then in each step there is a lot more. If you want to get a bit deeper you can follow this book of Andriy Burkov of Machine Learning engineering A couple notes in your ...
Carlos Mougan's user avatar
3 votes

Please review my sketch of the Machine Learning process

After 12 "Choose the model with highest scores." Maybe add "create ensemble of models" and try to improve accuracy further.
Sushil K's user avatar
3 votes

Can data leakage be sometimes acceptable?

If you have the whole population and do not want to predict anything, as stated in the question link you gave, then it is fine to use the whole population for preprocessing as it will give better ...
spectre's user avatar
  • 2,115
3 votes

Imputing time data for an event that hasn't occurred yet

It's very easy to over-complicate things. Agree with Broele that it depends. Let's look at what data you actually have: Since t0 (when you started collecting data), you know this location has not had ...
GooJ's user avatar
  • 435
3 votes
Accepted

Training a Model that Doesn't Always Have All the Features

You have three options in this situation: First option is to train separate models for all the possible combinations. This is a good option in case there are a lot of entries that do not have all the ...
Nemo_the_scientist's user avatar
3 votes

How to handle missing value if imputation doesnt make sense

Your approach of a binary categorical feature, is_married definitely sounds good. In some of my projects, I have checked for the percentage of missing values in a column. For instance, if a certain ...
g4th's user avatar
  • 131
2 votes

Missing Categorical Features - no imputation

Certain models are able to deal with missing values 'naturally', like certain tree based models. Most models however are just a mathematical function which is shaped after the training data. A very ...
Jan van der Vegt's user avatar
2 votes

Method for predicting price based on Geographical market, Product, and Company

It may be usefull to put what you are doing in a formal context, that way you can look at some standard solutions to your problem. You have obsvations of a (random) variable X (price), together with ...
VictorZurkowski's user avatar
2 votes

What would be the best way to impute data?

You can treat the missing feature as the target variable of a sub-problem and create a classifier (e.g., a linear model, SVM, etc) for it.
Ryan Zotti's user avatar
  • 4,149
2 votes

Imputation of missing values and dealing with categorical values

I think like @El Burro suggested, you I believe you should focus on feature transformation mainly. Use different techniques for different features. For straightforward features, such as occupation or ...
Bogas's user avatar
  • 586
2 votes

What is the difference between Missing at Random and Missing not at Random data?

Definitions of missingness process are tricky. Missing completely at random occurs when the missingness is really at random (MCAR; e.g. when conducting a survey there are error in the data entry ...
paoloeusebi's user avatar
2 votes

what to do if the missing data in one column is based on some value/condition in another column in r?

If you plan to use LightGBM or XGBoost, the advise is "Do not do anything". These methods treat NA in a specific way, different in each decision tree and the results obtained are usually much better ...
Grzegorz Sionkowski's user avatar
2 votes
Accepted

Imputation missing values other than using Mean, Median in python

So if you want to impute some missing values, based on the group that they belong to (in your case A, B, ...), you can use the groupby method of a Pandas DataFrame. ...
n1k31t4's user avatar
  • 14.9k

Only top scored, non community-wiki answers of a minimum length are eligible