When performing backpropagation with Adam algorithm, are the moment and the second moment of the weight vectors calculated also for the weights in hidden layers?


1 Answer 1


It looks like it.

The equations in the description of the algorithm in Hands-on Machine Learning as well as the original paper do not differentiate between parameters (weights) in different layers. Further, the scikit-learn implementation of Adam has ms and vs (first and second moment) vectors equal to the length of the parameters, and update these alongside updates to the weights themselves.

Typically, the parameters of a multilayer neural network are unpacked or unrolled into a single vector, which is what gets passed into the call to the optimizer function. Since the first and second moments are calculated for all of this vector in the scikit-learn implementation, you are right that the gradients are calculated for the weights in the hidden layers as well.

I hope that helps.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.