I have a classification problem where I want to find out whether feature engineering has improved my final model. Cross-validation is used evaluate the impact of the feature engineering steps, so there is no validation set (only train/test). In short, my situation entails the following:
- Collect data
- Train baseline model
- Feature engineering
- Train final model
- Compare final model against baseline (question)
Comparing the baseline and final models, I assume, can be done by running both models on the test set, subsequently evaluating the differences in their results (if any). However, I wonder if it is useful to compare the models using the training set as well/instead. It would be great if someone could elaborate on this issue.