Should the baseline comparison be done on the training or the test set results?

Question

I have a classification problem where I want to find out whether feature engineering has improved my final model. Cross-validation is used evaluate the impact of the feature engineering steps, so there is no validation set (only train/test). In short, my situation entails the following:

• Collect data
• Train baseline model
• Feature engineering
• Train final model
• Compare final model against baseline (question)

Comparing the baseline and final models, I assume, can be done by running both models on the test set, subsequently evaluating the differences in their results (if any). However, I wonder if it is useful to compare the models using the training set as well/instead. It would be great if someone could elaborate on this issue.

Answer 1

You definitely want the comparison to be based on the test set:

• Evaluating on the training set doesn't make sense for all the usual reasons.
• Especially in the case of different features, comparing the performance on the training set could be badly misleading: if one of the model overfits, its performance on the training set will appear better but its real performance (on the test set) is likely worse.

Note that it might make sense to study what happens on the training set (e.g. to measure overfitting), but that cannot be the real evaluation used for comparing the models.