3
$\begingroup$

I'm tuning the regularization parameter of a neural net (L2 regularization) using a grid. Starting with values 0.0005, 0.005, 0.05, 0.5, 5. Then if 0.005 brigs the best validation result, I continue using a grid like this: 0.0015, 0.0025, 0.0035, 0.0045, 0.0055, 0.0065, 0.0075, 0.0085, 0.0095. And it continues..

I would like to know if there's a more intelligent way to tune this parameter.

$\endgroup$
6
$\begingroup$

Wikipedia lists some well-known approaches to hyper-parameter searches.

The brute-force scan/search, or a grid search across multiple parameters, is still a very common and workable approach. As is random search, just trying some variations of parameters automatically and picking the best result from cross validation.

A guided search by intuitive feel of what might best work is also still quite common approach, despite being characterised as "optimisation by graduate descent" (a pun based on the fact that a senior researcher will hand off the tuning work to their smart junior researchers who make educated guesses on hyperparameter values) - for regularisation parameters you can look at training curves and difference between training loss vs cv loss to help decide where to look. You generally want to increase regularisation if there is a large difference and decrease it if the values become close but not very good. What counts and as "large difference" and "not very good" is subjective though, and typically you only get a feel for what works after a few random attempts and immersing yourself in the problem so that you recognise good vs bad behaviour of a model.

There are some claimed "smart" schemes that apply Bayesian optimisation or similar, and even some automated services or paid tools that attempt to make these available for tuning models automatically. However, I don't see these used much in practice.

Unless you are trying to win a Kaggle competition, you quickly hit diminishing returns attempting to fine-tune your parameter. I've rarely found tuning the second significant digit of a meta-param like L2 loss to be worth much attention. Maybe with the exception of testing between e.g. 1.0 and 1.5 - often useful to think in terms of geometric progression, not linear, even when fine-tuning.

You also have to be careful that you are not simply tuning to your cross-validation set, with no meaningful or justifiable improvement for test cases or in production. You can mitigate this in part by using k-fold cross validation (which can reduce error on metrics estimates effectively by using more data to estimate them), although of course that increases time/CPU costs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.