0
$\begingroup$

I'm reading the codebase of a project that uses Bidirectional-LSTM. The learning rate for it is 0.02. Later, someone improved the project by replacing LSTM with Roberta and decreased the learning rate to 5e-5. Why did they decrease the learning rate ?

$\endgroup$

1 Answer 1

0
$\begingroup$

The learning rate is a hyperparameter, and it has to be tuned for a specific neural architecture. Also, the optimal learning rate is affected by other hyperparameters, like the batch size.

We don't know why exactly the learning rate was changed in your case. It could be a proactive change due to the change in the architecture, or it could be reactive, i.e. it was lowered because the higher learning rate led to bad results after the architecture change.

$\endgroup$
2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.