I have a classification problem where I want to find out whether feature engineering has improved my final model. Cross-validation is used evaluate the impact of the feature engineering steps, so there is no validation set (only train/test). In short, my situation entails the following:

  • Collect data
  • Train baseline model
  • Feature engineering
  • Train final model
  • Compare final model against baseline (question)

Comparing the baseline and final models, I assume, can be done by running both models on the test set, subsequently evaluating the differences in their results (if any). However, I wonder if it is useful to compare the models using the training set as well/instead. It would be great if someone could elaborate on this issue.


1 Answer 1


You definitely want the comparison to be based on the test set:

  • Evaluating on the training set doesn't make sense for all the usual reasons.
  • Especially in the case of different features, comparing the performance on the training set could be badly misleading: if one of the model overfits, its performance on the training set will appear better but its real performance (on the test set) is likely worse.

Note that it might make sense to study what happens on the training set (e.g. to measure overfitting), but that cannot be the real evaluation used for comparing the models.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.