1
$\begingroup$

I’m trying to touch neural networks without using “in box” algorithms. And so I found out that nowhere is written how to calculate square of gradient for hidden layers in ADAM optimizer.enter image description here I took the description from original article and the problem is that Vt is some kind of normalizing coefficient, so it should be scalar. But for hidden layers gradient is not really a vector byt a matrix. So how should we calculate gt^2 in this case?

$\endgroup$
0
$\begingroup$

It is written right in the algorithm description that $g_t^2$ is elementwise square.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.