0
$\begingroup$

let's say I have a few linear layers $l_1 \dots l_n$:

$y=I(\dots I(IX + b_1) + b_2) \dots +b_n)$

where $n$ is sufficiently large and $I$ is the (nonparametric) identity matrix. The gradient for $b_n$ is then simply $\frac{\partial L}{y}$, but I am very confused about the gradients for the other biases. For example for $b_{n-1}$. Isn't it: $\frac{\partial L}{b_{n-1}}=\frac{\partial L}{l_{n-1}}\frac{l_{n-1}}{b_{n-1}}=(\frac{\partial L}{y} I^T)I=\frac{\partial L}{y}$?

But this would mean that by chaining $n$ bias terms I would get massive bias updates each gradient descent step! ($n$ times the usual bias update...). It also seems to violate the distributive-law$^1$?

$[1]$: What I mean: $y=I(\dots I(IX + b_1) + b_2) \dots +b_n) = X + \sum_{i=1}^n b_i = X + b$ for $\sum_{i=1}^n b_i =: b$. Shouldn't the gradient be compatible with this reshuffling?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.