I have an idea for a regularization-hyperparam selection method, which I haven't encountered before and can't find on Google, but I'm sure someone has already tried it and I'm wondering what are the best practices.

The most common method for hyperparam selection is to select different hyperparams (e.g some value for L2 regularization), train NNs with them, and test the NNs on some validation set - and select the best one. My idea is to train a single NN and test the NN on a validation set between epochs, and then auto-adjust the regularization hypeparam between epochs - if we see that the accuracy on the validation set is decreasing between epochs, then we should increase the value of the L1/L2/dropout. Naturally, this can be more efficient than training multiple NNs.

It's still a basic idea, and I'm sure it can be developed further. Is there research and best practices in this field?


As mentioned above you can try a couple of things, depending on which framework you are using and how you want to go with hyperparameter optimisation.

sklearn (with Keras wrappers):

  • GridSearchCV: slow but sure to find optimal hyperparams
  • RandomizedSearchCV: faster and almost as good as GridSearch

Keras (with Keras Tuner):

  • BayesianOptimization: tuning with Gaussian process
  • Hyperband: Variation of HyperBand algorithm (Li, Lisha, and Kevin Jamieson. Journal of Machine Learning Research 18 (2018): 1-52.)

Pytorch (with RayTune):

  • Population Based Training: trains a group of models (or agents) in parallel (https://deepmind.com/blog/population-based-training-neural-networks)
  • HyperBand: as above
  • ASHA:Compared to the original version of HyperBand, this implementation provides better parallelism and avoids straggler issues during elimination
  • Median Stopping Rule: implements the simple strategy of stopping a trial if its performance falls below the median of other trials at similar points in time.
  • FIFOScheduler: Simple scheduler that just runs trials in submission order
  • $\begingroup$ I'm familiar with these solutions, but thats not what i'm asking.. I'm asking about something similar to the solution I proposed - has anyone tried something like it? because I can see how it would be more efficient than many of these solutions (as I said in the post - if only involves training one NN as opposed to grid search) $\endgroup$ – Oren Matar Sep 15 '20 at 8:55
  • $\begingroup$ Are you? You suggest training a single NN and update hyperparameters between epochs which is not ideal as it stands given grad and hyperparameter values updates would intermix between epochs. I believe what you are conceptually looking for is approiaches like hyperopt and PBT. That is why i posted those above. I understand the answer is not useful though so will delete shortly. Many thanks $\endgroup$ – hH1sG0n3 Sep 15 '20 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.