2
$\begingroup$

Given, we have a learning rate, $\alpha_n$ for the $n^{th}$ step of the gradient descent process. What would be the impact of using a constant value for $\alpha_n$ in gradient descent?

$\endgroup$
  • $\begingroup$ Do you mean a constant value of $\alpha$ for each step? $\endgroup$ – Wes Feb 11 '19 at 21:18
5
$\begingroup$

Intuitively, if $\alpha$ is too large you may "shoot over" your target and end up bouncing around the search space without converging. If $\alpha$ is too small your convergence will be slow and you could end up stuck on a plateau or a local minimum.

That's why most learning rate schemes start with somewhat larger learning rates for quick gains and then reduce the learning rate gradually.

| improve this answer | |
$\endgroup$
1
$\begingroup$

Gradient descent has the following rule:

$\theta_{j} := \theta_{j} - \alpha \frac{\delta}{\delta \theta_{j}} J(\theta)$

Here $\theta_{j}$ is a parameter of your model, and $J$ is the cost/loss function. At each step the product $\alpha \frac{\delta}{\delta \theta_{j}} J(\theta)$ gets smaller as we get closer to the gradient $\frac{\delta}{\delta \theta_{j}} J(\theta)$ converging to 0. $\alpha$ can be constant, and in many cases, it is, but varying $\alpha$ might help converge faster.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.