# Robustness of hyperparameter tuning

I use a Bayesian hyperparameter (HP) optimization approach (BOHB) to tune a deep learning model. However, the resulting model is not robust when repeatedly applied to the same data. I know, I could use a seed to fix the parameter initialization, but I wonder if there are HP optimization approaches that already account for robustness.

To illustrate the problem, let's consider a one-layer neural network with only one HP: the hidden size (h). The model performs well with a small h. With a larger h, the results start to fluctuate more, maybe due to a more complex loss landscape; the random initialization of the parameters can lead to a good performance, or to a very bad performance if the optimizer gets stuck in a local minimum (which happens more often due to the complex loss landscape). The loss vs h plot could look something like this:

I would prefer the 'robust solution', while the 'best solution' is selected by the HP optimizer algorithm. Are there HP optimization algorithms that account for the robustness? Or how would you deal with this problem?

As I understand them, Bayesian optimization approaches are already somewhat robust to this problem. The evaluated performance function is usually(?) considered noisy, so that the search would want to check nearby the "best solution" $$h$$ to improve certainty; if it then finds lots of poorly performing models, its surrogate function should start to downplay that point. (See e.g. these two blog posts.)

If the instability is large due to random effects (e.g. initializations of weights that you mention), then just repeating the model fit and taking an average (or worst, or some percentile) of the performances should work well. If it's really an effect of "neighboring" $$h$$ values, then you could similarly fit models near the selected $$h$$ and consider their aggregate performance. Of course, both of these add quite a bit of computational expense; but I think this might be the closest to "the right" solution that doesn't depend on the internals of the optimization algorithm.

• Thanks. Running the model multiple times as you suggest is not feasible, unfortunately. Do I understand correctly that not the actual scores are taken but the surrogate model fit? I thought the surrogate model is just used to determine next sampling points. – bask0 Jul 9 '20 at 7:35
• You're right that the actual performance is considered for reporting the best score, not the surrogate (although you may be able to post-process the search results to do so!). In fact, if the performance around that point is particularly noisy, your search may come up with other deep wells, so that the search focuses even more on that location, in an attempt to reduce the uncertainty near these points of high-on-average performance... – Ben Reiniger Jul 9 '20 at 15:39
• Maybe you can hack your objective function so that it averages other nearby results, as in my second paragraph, but using cached values from existing fits... the list of results would have to dynamically change though, which would probably cost a lot of re-computation of the surrogates... – Ben Reiniger Jul 9 '20 at 15:43
• Which brings me back to post-processing the search results. The main downside there is that your search has (potentially) wasted time exploring a region you'll ultimately deem too variable for selection. Sorry to be throwing out half-formed ideas, but maybe something will inspire someone with a better strategy. – Ben Reiniger Jul 9 '20 at 15:46
• Hey Ben. I fully agree with your comments. It seems to be quite a hack to get something like this to run. Maybe I should just invest my time into getting a less noisy model... I will mark your comment as answer, thanks! – bask0 Jul 14 '20 at 15:50

One option is not to measure the performance of the hyperparameters on the loss function of the training data but measure performance of the hyperparameters on the elevation metric on the validation data. The end goal of the most machine learning systems is the ability to predict on unseen data. Focusing on "best solution" as measured by loss function on training will lead to overfitting / non-robust solutions.

• This doesn't seem to answer the question; does OP state somewhere that the cartoon graph isn't already the validation score? – Ben Reiniger Jul 6 '20 at 23:22
• Yes, that's actually correct. The problem can also occur when using the validation set score (which I do). – bask0 Jul 7 '20 at 8:21