why should i do target encoding within cv loop?

Question

i wish to use target encoding, using the category encoders sklearn library. I don't really understand why it is necessary to include this as a step in a sklearn pipeline WITHIN the cross validation loop?

e.g. this example here does so Target encoding with KFold cross-validation - how to transform test set?

my methodoloigy is similar to the one in the link except i do not use any smoothing. my dataset is quite large around 300-500k.

however looking at my result below does this not indicate over fitting?

Answer 1

From the performance point of view, you can one-hot encode the whole dataset before the model fitting and you will get identical results.

Why people do it in the CV loop?

Ideological perspective In the CV, we set the part of the data aside and run the model on the given dataset from scratch. For example, you may want to normalize the database, but minimum, maximum, mean, and std are dependent on a particular fold, and, therefore, should be done inside the CV loop. Therefore, preprocessing is inside the CV loop, and so one hot encoding belongs here.

Sparse categorical variables Suppose that the output of your model is a categorical random variable with tons of categories. For example, your output is a single word that described the image, and there are thousands of classes. Instead of one-hot encoding and getting tons of columns of almost all zeros, you may want to use sparse categorical variables, i.e., to map the target or a feature to integer values. For simplicity of the argument, suppose that a categorical feature is encoded as a sparse variable. Then, the values of the encoded column are 0,1,2...,n and for the linear regression, the order matters. In other words, 2 is closer to 0 than to 100. To make sure that your model performs well regardless of encoding order, you need to check cross-validation.