The way I read almost lots of ML advice on these datasets sounds like "You train a model that's randomly chosen hyperparameters first on the training set, then you ignore this bit of the work, and hyperparameter tune on a validation set, then choose the final model you want and test it on the test set". This doesn't make sense, but I swear I read something like this quite a bit.

But likewise hyperparameter tuning is expensive. You can end up with many (e.g. "N" models all of different hyperparameters). If we train them all on the training set, then that is super expensive (biggest dataset training times 'multiplied by' N models). Then we could check them on the validation set. If we just "choose the best performance" and that's it, and then don't do any new training this version of hyperparameter tuning seems expensive, and I don't understand why we'd need a test set.

Other options involve seemingly merging the train and val sets, to do a final round of tuning.

I've no real idea what's going on.

Can someone explain hyperparameter tuning steps in simple ways, and how the datasets are used? Nothing seems to make sense to me.


1 Answer 1


You are asking about a model's ability to generalize to unseen examples.

Consult Abu-Mostafa, et al., Learning From Data, § 5.3 Data Snooping.

If a data set has affected any step in the learning process, its ability to assess the outcome has been compromised.

There is a giant space of models that you might choose from, based on your knowledge of the underlying generative process (e.g. "sales of widgets"). You might choose a linear model for regression, or a quadratic or other polynomial model, or perhaps SVM, random forest, or a multitude of other hypothesis families and loss functions.

Having decided on a family of hypotheses, you next must settle on the hyperparameters that specify a particular hypothesis, such as "max tree depth" for RF, together with per-node learned parameters. Commonly there will be more than half a dozen hyperparameters. Their values matter, they affect model performance.

So we can sweep through a grid of proposed hyperparameter settings, or better we can quickly explore hyperparameter space by choosing random points.

The reason we do this is so we can generalize. To get 100% accuracy on already-seen training data is trivial; any RDBMS could parrot back Y for such X values. But for novel X values, being able to interpolate or otherwise generalize requires a more sophisticated approach. The generative process in the world produced observables that adhere to some pattern, and it is the model's job to learn that pattern. For example, proposing "interpolation" suggests a continuous smoothly differentiable generating function, to which we can usefully apply interpolation techniques.

no cheating

We evaluate our model's ability to generalize to unseen data by scoring it against unseen test data, which superficially resembles scoring against folds of training data.

If any of the model's parameters or hyperparameters have settings based on some test data, then that data cannot be used to assess the model's ability to generalize to unseen data. We need new, unseen data to accomplish such an assessment.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.