1
$\begingroup$

When doing hyperparameters optimisation, like a Random Search, should you add a search space for the learning rate ?

My intuition is that some HP might work better with a certain LR, and be sub-optimal with a lower LR. But if I add LR to the search space, I fear that the random search will only favour high LR trials, as they will reach lower loss for the same limited number of max epochs.

What would be the right way to do it ?

$\endgroup$

1 Answer 1

0
$\begingroup$

Learning rate probably should not be considered an independent hyperparameter as it is usually a good idea to adjust it proportionally to batch size.

$\endgroup$
2
  • $\begingroup$ That's interesting, thank you ! In the same way, can we not add the batch size as an hyperparameter ? $\endgroup$ Oct 7, 2021 at 15:39
  • $\begingroup$ @M.Garrigues Anything you have to define manually (meaning: not trainable) can be considered a hyperparameter. For the case of batch size/learning rate, they probably should be considered as a single hyperparameter together. But compared to adjusting the number of neurons or layers, their effect would be small. $\endgroup$
    – serali
    Oct 7, 2021 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.