1
$\begingroup$

From what i have learned, people say that it is more correct if i preprocess data after splitting it to train/test dataset. My questions are

1.Does it mean we detect flaws of the data + preprocess it after the splitting? if yes/no, why?

2.Is it okay if i detect flaws of the data before splitting it and then preprocess the flaws after splitting it? if it is/ no it is not, why?

$\endgroup$

1 Answer 1

2
$\begingroup$

The goals of all these methodological guidelines is to avoid data leakage.

Example: let's imagine we want to classify short messages (e.g. tweets). When inspecting the data we find various kinds of smileys: :-), :|, :-/... At preprocessing stage we replace all smileys found in the data with a special token like <smiley> (or something more specific).

  • If the detection/replacement is done on the whole data, every occurrence of a smiley in the test set is replaced with <smiley>.
  • If the detection/replacement is done on the training set, even after preprocessing there might be a few smileys left in the test, because some uncommon ones didn't appear in the training set.

In the first case there is data leakage: we fixed some issues in the test set even though this wouldn't have been possible with actual fresh data (here the variants of smiley that were not seen in the training set). In the second case the test set is "imperfect", i.e. it's exactly as if it was made of "fresh" unseen data, therefore the evaluation will be more realistic.

This example shows why it's always safer to separate the data first, design the preprocessing steps on the training data, then apply exactly the exact same preprocessing steps to the test data.

In practice there can be cases where it's more convenient to apply some general preprocessing to the whole data. The decision depends on the task and the data: sometimes the risk of data leakage is so small that it can be neglected. However it's crucial to keep in mind that even the design of the preprocessing can be a source of data leakage.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.