Assuming a standard train/validation/test split, the common practice is (a) to train multiple models with different hyper-parameter configurations on the training set, (b) to evaluate these models only on the validation set and (c) retrain the optimal configuration on the combined (train+validation) set and make unbiased evaluation on test set. It is the second step that concerns me. A similar practice is recommended for cross-validation, in which only the CV validation error is used to select the hyper-parameter configuration. Even for scikit-learn, the implemented GridSearchCV and RandomSearchCV approaches consider only validation loss performance by default (although it is possible to retrieve train loss estimates).

This common practice has been question elsewhere e.g., https://stats.stackexchange.com/questions/425503/cross-validation-train-and-test-error. Also, there have been several posts on the issue of hyper-parameter overifitting caused by focusing only on the validation set during hyper-parameter optimization (HO), and there seems to be a lot of "wishful thinking" that it won't happen. Andrew Ng has even written a paper on why it is not optimal to select hyper-parameters based only on validation error: https://ai.stanford.edu/~ang/papers/cv-final.pdf

Assuming also a scenario in which the training set is much larger than the validation set, should we not be considered about the performance of the model on a large number of examples? A related concern is quoted from Dikran Marsupial https://stats.stackexchange.com/a/551945/307304:

One thing that is not widely appreciated is that over-fitting the model selection criteria (e.g. validation set performance) can result in a model that over-fits the training data or it can result in a model that underfits the training data.

It is however true that it is also a common practice to compare validation and training loss (to see if the model is over/under fitting), but only after HO. This bring's forth the following questions:

(1) Why do we not compare/combine train loss with validation loss during HO to find configurations that simultaneously achieve low validation error AND similar validation and training performance? In other words, to search directly for an accurate model that is not over-fitting/under-fitting. Of the top of my mind, find below some examples of alternative optimization targets that could be used instead of pure validation performance.

$$0.8*loss_{val} + 0.2*loss_{train}$$
$$\frac{loss_{val}+loss_{train}}{2} - \frac{|loss_{val} - loss_{train}|}{2}$$

(2) If none of the above, are there other canonical and widely accepted ways of integrating both training and validation in the hyperparameter optimization process? If not, why is this not a common approach?

In conclusion, the idea can be summarized as an optimization methodology that presents some symmetry: "if validation set can be used to avoid parameter over-fitting on the training set, then the training set could be used to avoid hyper-parameter over-fitting on the validation set".

  • 2
    $\begingroup$ There is statistical precedent for this, e.g. the "one standard error rule" in lasso regression, see e.g. stats.stackexchange.com/q/80268/232706 and stats.stackexchange.com/q/138569/232706. That doesn't carry over easily because there isn't a total order on most hyperparameter spaces for "simplicity". // I've seen something like your proposed "overfitting penalties" used before, but I don't have anything solid on how effective they are. $\endgroup$
    – Ben Reiniger
    Jan 24 at 15:33
  • $\begingroup$ Thank you. So you mean that the "one standard rule" only makes sense when using lasso regression, but not for other learning methods? $\endgroup$
    – Enk9456
    Jan 25 at 11:34
  • 1
    $\begingroup$ It says to pick the "simplest" model within one standard error of the optimal score. In lasso, "simplest" means highest penalty and fewest nonzero coefficients. If you're tuning one hyperparameter of a GBM, you could probably do the same. But if you're tuning lots of GBM parameters, which combination is "simpler" isn't so easy. I suppose you could again look to the train-test score difference instead of the nature of the hyperparameters; has that been done anywhere?... $\endgroup$
    – Ben Reiniger
    Jan 25 at 13:38
  • $\begingroup$ Alright, thank you for this interesting idea. Optimizing for a model that performs well on the validation set but is also simple in terms of hyper-parameters seems like a good alternative to using the training loss. That would be a form of "hyper-parameter regularization" which is not the same as the usual "parameter regularization" e.g., lasso, dropout etc. The only requirement is to define a function that estimate the "complexity" of the current configuration. Regarding your question, I 've not seen anyone do that "during" HO, only after to check if there is overfitting/underfitting. $\endgroup$
    – Enk9456
    Jan 25 at 15:06

1 Answer 1

  1. The simple answer might be that we want our model to be the best in a real-world application. Because of that, in most cases we don't care about training loss as it's not a good sample of real-world data (due to the possible overfitting) and validation loss is - validation data are not directly known by the model. On the other hand, a huge gap between training and validation loss is a disturbing phenomenon and might testify against the quality of the model.
  2. If you want to incorporate training loss into your HO, still there might be approaches when it would be beneficial. It might be a plus in Bayesian Optimization - you can read more about that in this well-cited work.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.