1
$\begingroup$

I am looking up sources to implement SGD with momentum, but they are giving me different equations.

(beta is the momentum hyper-parameter, weights[l] is a matrix of weights for layer l, gradients[l] are the gradients for layer l, etc)

This source gives:

v[l] = beta*v[l] - learning_rate*gradients[l]
weights[l] = weights[l] + v[l]

But this source gives:

v[l] = beta*v[l] + learning_rate*gradients[l]
weights[l] = weights[l] - v[l]

Are they equivalent?

Also, does it matter if beta + learning_rate != 1? (In this case this would be different from the exponential moving average equation, where they sum to 1).

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.