Data Leakage possibility related to replacing rare values on a categorical column

Question

I am exploring the possibility of data leakage for categorical columns, replacing rare values with category 'Other'

Let's say I have a DF with 40 categorical columns. I will check each of them, find frequencies of unique values, and If a value is less frequent than 1%, I will replace it with category value of 'Other.

Same process will be applied to each column, most probably with a loop.

Actually I am highly suspicious of data leakage, when applying this directly to whole data. In the process, I will apply a CV scheme. And I am also considering to put it into preprocessing pipeline, calculate rare categories only with Train folds, each time. While training the production model at the end, I can use whole data of course.

I want to ask, Is it overkill to put this replacement into CV pipeline, and apply several times for only Training data? (Since I have 40 categorical columns, it may mean 40*N times of calculation, for N fold CV)

If I apply it to whole data at the beginning, will I be prone to data leakage?

Thanks.

Answer 1

The short answer is yes.

The long answer is the following. When computing statistics or similar estimates, they should be computed only on the training data. Why? Because you should never poison your training data with information from test data since the basic assumption is that the latter is unavailable during training. As a toy example, if you want to min-max normalize your data before training, the min and max are to be computed only on the training data. Then, you train your model, say, $f$. Finally, when testing $f$, the normalized features should use the min and max that you have computed before training; otherwise, you most likely will have a data leakage. Thus, the same reasoning applies to your issue. In particular, it is not an overkill; you are simply evaluating your model, which is always a good thing to do, even if it overkills you by long-hours computation.