let's say I have a few linear layers $l_1 \dots l_n$:

$y=I(\dots I(IX + b_1) + b_2) \dots +b_n)$

where $n$ is sufficiently large and $I$ is the (nonparametric) identity matrix. The gradient for $b_n$ is then simply $\frac{\partial L}{y}$, but I am very confused about the gradients for the other biases. For example for $b_{n-1}$. Isn't it: $\frac{\partial L}{b_{n-1}}=\frac{\partial L}{l_{n-1}}\frac{l_{n-1}}{b_{n-1}}=(\frac{\partial L}{y} I^T)I=\frac{\partial L}{y}$?

But this would mean that by chaining $n$ bias terms I would get massive bias updates each gradient descent step! ($n$ times the usual bias update...). It also seems to violate the distributive-law$^1$?

$[1]$: What I mean: $y=I(\dots I(IX + b_1) + b_2) \dots +b_n) = X + \sum_{i=1}^n b_i = X + b$ for $\sum_{i=1}^n b_i =: b$. Shouldn't the gradient be compatible with this reshuffling?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.