Skip to main content
11 votes
Accepted

How high of a correlation coefficient of a feature with a target variable is considered too high?

You should not remove features just because their correlation to the target is high. Such high correlation is a sign of potential target leakage. You should understand why their correlation is high. ...
noe's user avatar
  • 27k
7 votes
Accepted

Does label encoding an entire dataset cause data leakage?

The cleanest solution would be to apply scikit's OneHotEncoder with the handle_unknown parameter set to "ignore": ...
Jonathan's user avatar
  • 5,430
7 votes

Does label encoding an entire dataset cause data leakage?

Encoding labels before splitting the data set should not cause leakage, particularly in the case of ordinal encoding. Ordinal encoding is just a transform from "label space" to "...
zachdj's user avatar
  • 2,734
4 votes
Accepted

How to keep the test data from leaking into the training process of a machine learning algorithm?

Just to clarify (and I think you've got this right, but I'm just being careful), it is best practice to: 1: Split your data into train and test 2: Split train into train and eval 3: Grid search ...
gazza89's user avatar
  • 266
4 votes
Accepted

Stacking: Use predictions of train or test to create features for level 1 classifier

I'm not sure if there's any standard about this, but I usually proceed by splitting the training set into two parts A and B: A is used as training set for level 0 models B is used as test set for the ...
Erwan's user avatar
  • 25.6k
3 votes

What can I do when my test and validation scores are good, but the submission is terrible?

Imho the most likely explanation is that the submission test set doesn't follow the same distribution as the training/validation/test data that you used to train and evaluate the model. In other words ...
Erwan's user avatar
  • 25.6k
3 votes
Accepted

Splitting before tfidf or after?

You should split before tf-idf. If you learn tf-idf also on the test set you will have data leakage. In example, you won't have out-of-vocabulary words in inference on the test set, what might happen ...
Amit Keinan's user avatar
3 votes

Can data leakage be sometimes acceptable?

If you have the whole population and do not want to predict anything, as stated in the question link you gave, then it is fine to use the whole population for preprocessing as it will give better ...
spectre's user avatar
  • 2,125
3 votes
Accepted

Can preprocessing the whole population cause data leakage?

If you have the entire population, there is no need for inference. Thus data leakage is not an issue. You can fit any transformation on the data without a concern for its effect on prediction because ...
Brian Spiering's user avatar
3 votes
Accepted

Manual feature engineering based on the output

You can create features based on output values, but you should be careful in doing this. When you use the value of class_city (based on percentage of signed for that city) for a given data point, ...
raghu's user avatar
  • 651
3 votes

Manual feature engineering based on the output

No you should not do this, it is causing a data leak. Data leaks happen when the data you are using to train a machine learning algorithm happens to have the information you are trying to predict. ...
Simon Larsson's user avatar
3 votes

How does a data leakage work?

My reccomendation will be to see what are the important features of the model. Probably use SHAP if you are on random forest https://github.com/slundberg/shap You might see that one features is ...
Carlos Mougan's user avatar
3 votes

Is it unethical to gather data from data leaks about demographics?

Even if no personally identifiable information is revealed about individuals, collecting data from data leaks can still raise ethical and legal concerns. This is because such data is often sensitive ...
RegressIt's user avatar
  • 415
2 votes

Is normalizing the validation set of time series a kind of look ahead bias?

To block the data leakage from the validation set to the training set in step (2), We should first split the data to training and validation sets, then Calculate the mean and standard deviation only ...
Esmailian's user avatar
  • 9,362
2 votes
Accepted

How to deal with possible data leakage in time series data?

There is no need for sample tests. A customer may have received many loans 1 to n - 1. To predict the default rate of nth request at time t(n), you are allowed to use any information up until t(n). ...
Esmailian's user avatar
  • 9,362
2 votes

What is the difference between data leakage and endogeneity?

Endogeneity refers to explanatory variables correlated with error term because of a missing variable or measurement error. Data Leakage is introduction of spurious explainability in the model because ...
Sunny's user avatar
  • 31
2 votes

classification feature selection

You have a problem of data leakage. The "days since invitation was sent" feature contains all the information on the concept. Therefore, adding it as a feature will prevent most common classifiers ...
DaL's user avatar
  • 2,643
2 votes
Accepted

Is there potentially data leakage during imputation for time-varying sensor data?

If you are using information from the future to impute missing data would be data leakage as you would not have this extra information when the model is in production and trying to predict future ...
Oxbowerce's user avatar
  • 7,657
2 votes
Accepted

Information leakage when train/test are truly i.i.d.?

Well, keep in mind that when you standardize/impute data you're estimating parameters. Given the conditions that you've defined and having enough data such that the estimates are good, then I don't ...
David Masip's user avatar
  • 6,101
2 votes
Accepted

Why leaky features are problematic

Yes, the argument you are giving is perfectly valid. But Let look at two different scenarios and see how it does not benefit in real world: If you already have a variables which perfectly predicts ...
Ashwiniku918's user avatar
  • 2,024
2 votes
Accepted

Avoid leakage in NLP extraction

It is best practice to split the data into train and test datasets. Make modeling choices only on the train data set. Evaluate the usefulness of those choices on the test dataset. Traditional NLP ...
Brian Spiering's user avatar
2 votes

what qualifies as a data leakage?

One definition of data leakage is providing the model with data during training that would not be available at a future prediction time. The variable "total target achieved/units purchased as on ...
Brian Spiering's user avatar
2 votes
Accepted

what qualifies as a data leakage?

Data leakage occurs in cases when you train a model with data that is not available for future testing/inference; or when you use same piece of data for training, and then for validation and/or ...
Stefan Popov's user avatar
2 votes

What is the best way to avoid data leaking in timeseries forecasting multiple labels?

There are few strategies that you could use to avoid data leakage: Use Cross-validation: For the time series data, use TimeSeriesSplit from ...
Harshad Patil's user avatar
1 vote
Accepted

How to fit Word2Vec on test data?

Your dataframe new is already the correct embeddings to use for the test set. Just tokenize the test reviews, limit to those words in your training vocabulary, and ...
Ben Reiniger's user avatar
  • 12k
1 vote
Accepted

Data Leakage when preprocessing categorical features?

Think of the data being organized in rows (instances) and columns (features). Any pre-processing step which mixes information between/across rows can lead to data leakage. A typical example is ...
Jonathan's user avatar
  • 5,430
1 vote

Does binning a time series with pd.qcut (using quantiles) create data leakage?

I think that indeed you may have leakage by using pd.qcut. A solution to avoid that leakage is to do it in a time-series cross-validation fashion. The idea is to ...
David Masip's user avatar
  • 6,101
1 vote

K-Fold cross validation and data leakage

A reproducible example with no data leakage: In there I'm scaling the data only with the train data on the k-fold stage ...
Multivac's user avatar
  • 3,009
1 vote

How to split up my dataset in a train and testset, in order to prevent data leakage?

In multi-patient datasets a typical cross validation split is leave-subject-out, check this out: What is difference between leave one subject out and leave one out cross validation Basically, each ...
LuckyLuke's user avatar
  • 146
1 vote
Accepted

Is it right to maintain the train distribution in test set for unbalanced data?

If the training set was unbalanced the chances are the model will be biased. Not really. Depending on the loss function you use. Also, note that for data to be unbalanced at least it has to be in a ...
Carlos Mougan's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible