2
$\begingroup$

This is a general question which often comes up when tuning deep learning and machine learning algorithms such as recurrent neural network, multilayer perceptron or SVM etc.

When we tune the hyper parameters of a deep learning model every possible combination of hyper parameters results in a different model. And we select an optimal combination based on the loss curves. What exactly is an optimal combination of hyper parameters?

My question is exactly this: There is an infinite number of combinations of hyperparams possible. We know that there are many possible configurations of hyperparams possible that give similar generalization error. What should the model selection decision be based upon? And how do I know I have hit the bottom and no other combination of hyperparams will give me better results?

| improve this question | |
$\endgroup$
  • $\begingroup$ Well the answer is pretty simple, we keep on exploring as much as we can but we do keep a set of values which currently does the job well but that doesn't mean there aren't better parameters than the current one.. $\endgroup$ – Aditya Jun 10 '18 at 10:59
  • $\begingroup$ The best combination is the one which optimizes our objective, which is typically to predict unseen data. And it is in general impossible to know whether or not we have found this optimum point except in the most trivial cases. We just do the best we can and hope for a reasonable approximation. $\endgroup$ – dsaxton Jul 10 '18 at 22:42
1
$\begingroup$

"What exactly is an optimal combination of hyper parameters?" The combination of hyperparameters that produces the lowest possible error on unseen data for that model architecture.

"What should the model selection decision be based upon?" The best way to estimate error on unseen data that I know of is k-fold cross-validation. I examine the mean and standard deviation of the error across the k-folds; in most cases, I select the model with the smallest standard deviation among the models with the best mean error.

"And how do I know I have hit the bottom and no other combination of hyperparams will give me better results?" As far as I know, one can never know what combination of hyperparameters will produce the lowest possible error on unseen data. In my experience, a good search strategy will get you close enough to the optimal combination so that further search is not worth the effort.

| improve this answer | |
$\endgroup$
0
$\begingroup$

This comes down to "how can we be sure we've found global minima, what if it's just a few steps away".

Until we go there, it's unknown. However, there is a clever way to be very sure we've found a global minimum. I am too unexperienced to understand it, but here it is (Tensor Methods: A new paradigm for training probabilistic models and for feature learning, Anima Anandkumar) https://www.youtube.com/watch?v=B4YvhcGaafw

As I recall, they "un-bend the search space" so it literally exposes the global minimum, then just select it ...ugh :s

If someone could comment on my understanding of the video, I would be thankful

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.