0
$\begingroup$

One of the assumptions for finding good hyperparameters using Bayesian optimization (GP) is that the unknown function is smooth. Is this assumption valid for neural networks or at least for most of the neural networks? Can we find any reference?

$\endgroup$
1
$\begingroup$

Neural networks are optimized by gradient descent which assumes the loss function for parameters is a differentiable function, in other words smooth. Given the nature of the differentiable loss function, Bayesian Optimization could be used for neural networks hyperparameter optimization. In fact, gradient descent can be used to learn the hyperparameters themselves as evidenced in the paper - "Gradient Descent: The Ultimate Optimizer".

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.