1
$\begingroup$

Given a Computation Graph with a node (like the one below), I understand that I can use the upstream gradient dL/dz to calculate all of my downstream gradients.

But what if there are multiple upstreams? Would the final upstream gradient just be the sum of the derivatives? Or the product? Or the mean? Or something else?

For example, instead of just 1 upstream: [z], we are given 3 upstreams: [z0, z1, z2]:

dL/dz = dL/dz0 + dL/dz1 + dL/dz2          // sum?
dL/dz = dL/dz0 * dL/dz1 * dL/dz2          // or product?
dL/dz = (dL/dz0 + dL/dz1 + dL/dz2) / 3    // or mean?

gradients of a node in a computation graph

Improve this question
$\endgroup$
2
  • $\begingroup$ I think it is the sum but not sure... $\endgroup$
    – Ibrahim
    Commented Jan 22 at 0:07
  • $\begingroup$ After some testing, I have found that the sum is the correct answer and I have answered my own question below. $\endgroup$
    – Ibrahim
    Commented Feb 4 at 15:34

1 Answer 1

Reset to default
0
$\begingroup$

In a typical Computational Graph Node, its "Final Upstream Gradient" (dL/dz in the above case) is equal to the sum of all n upstream gradients (z0, z1, z2 in the above case):

dL/dz = dL/dz0 + dL/dz1 + ... + dL/dzn
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.