This is a general question which often comes up when tuning deep learning and machine learning algorithms such as recurrent neural network, multilayer perceptron or SVM etc.
When we tune the hyper parameters of a deep learning model every possible combination of hyper parameters results in a different model. And we select an optimal combination based on the loss curves. What exactly is an optimal combination of hyper parameters?
My question is exactly this: There is an infinite number of combinations of hyperparams possible. We know that there are many possible configurations of hyperparams possible that give similar generalization error. What should the model selection decision be based upon? And how do I know I have hit the bottom and no other combination of hyperparams will give me better results?