I am a newbie to this stuff so I am sorry if my question is stupid~

I need help understanding what data leakage between X_train and X_test is and when exactly it happens. I am currently working on a dataset where I am using KNN imputer to fill in the missing values. I need to scale the data for knn imputation and I am doing the train-test-split and applying machine learning models after the imputation process. I read that data leakage might happen during scaling so we should scale after splitting, fit_transform the train set, and only transform the test set. I am unsure about how that would work in my case since I am scaling the data to be able to impute for missing values and I actually reach the train-test-split stage later. Should I be worried about data leakage the way I am doing things?

Here is the code:

enter image description here

Although here I am doing the splitting + applying DT algorithm right after imputation, I have other steps like feature selection left so I won't reach the train-test-split and decision tree part until much later.

  1. split the data into train and test
  2. fit your imputer based on the train data set (use just fit)
  3. use the fitted imputer and fill missing value in the training dataset (use transform)
  4. Train you decision tree based on the training dataset
Now you are done with the training step. start testing a follows
  1. Use imputer trained in step 2 and transform function to replace missing values in the test dataset
  2. Use the trained decision tree for test prediction and evaluating the performance of your model on the unseen test dataset

The data should be split at the beginning. As the name of "unseen" shows, we should not use any information from the test dataset when we are training the model; otherwise, it is data leakage.

  • $\begingroup$ Thank you for answering! what if this was the order I was following:- 1. read in the data, 2. exploration, 3.outlier/noise handling, 4. encoding, 5.Scaling + KNN imputation for missing values, 6. Feature Selection/PCA analysis, 7. Apply machine learning algorithms, 8. Evaluation of models. Would I do the train test split right before the scaling+ imputation step or right after reading in the data? That way, which steps would I repeat for the test data? $\endgroup$
    – moii789
    Jun 3 '20 at 17:15
  • $\begingroup$ I added a few lines to my response! $\endgroup$
    – nimar
    Jun 3 '20 at 18:04

Data leakage occurs whan you have in your training set information which you are not supposed to have. This might be, for example, having in the training set information contained in the test set. Generally speaking, data leakage can provide a model which is "too good to be true", but not to be trusted as it was build using "cheatful" information. Find more information here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.