Gradients for bias terms in backpropagation

Question

I was trying to implement neural network from scratch to understand the maths behind it. My problem is completely related to backpropagation when we take derivative with respect to bias) and I derived all the equations used in backpropagation. Now every equation is matching with the code for neural network except for that the derivative with respect to biases.

z1=x.dot(theta1)+b1

h1=1/(1+np.exp(-z1))
z2=h1.dot(theta2)+b2
h2=1/(1+np.exp(-z2))

dh2=h2-y
#back prop

dz2=dh2*(1-dh2)
H1=np.transpose(h1)
dw2=np.dot(H1,dz2)
db2=np.sum(dz2,axis=0,keepdims=True)

I looked up online for the code, and i want to know why do we add up the matrix and then the scalar db2=np.sum(dz2,axis=0,keepdims=True) is subtracted from the original bias, why not the matrix as a whole is subtracted. Can anyone help me to give some intuion behind it. If i take partial derivative of loss with respect to bias it will give me upper gradient only which is dz2 because z2=h1.dot(theta2)+b2 h1 and theta will be 0 and b2 will be 1. So the upper term will be left.

b2+=-alpha*db2

Answer 1

The bias term is very simple, which is why you often don't see it calculated. In fact

db2 = dz2

So your update rules for bias on a single item are:

b2 += -alpha * dz2

and

b1 += -alpha * dz1

In terms of the maths, if your loss is $J$, and you know $\frac{\partial J}{\partial z_i}$ for a given neuron $i$ which has bias term $b_i$ . . .

$$\frac{\partial J}{\partial b_i} = \frac{\partial J}{\partial z_i} \frac{\partial z_i}{\partial b_i}$$

and

$$\frac{\partial z_i}{\partial b_i} = 1$$

because $z_i = (\text{something unaffected by } b_i) + b_i$

---

It looks like the code you copied uses the form

db2=np.sum(dz2,axis=0,keepdims=True)

because the network is designed to process examples in (mini-)batches, and you therefore have gradients calculated for more than one example at a time. The sum is squashing the results down to a single update. This would be easier to confirm if you also showed update code for weights.

Answer 2

I would like to explain the meaning of db2=np.sum(dz2,axis=0,keepdims=True) as it also confused me once and it didn't get answered.

The derivative of L (loss) w.r.t. b is the upstream derivative multiplied with the local derivate: $$ \frac{ \partial L}{\partial \mathbf{b}} = \frac{ \partial L}{\partial Z} \frac{ \partial Z}{\partial \mathbf{b}} $$

If we have multiple samples Z and L are both matrices. b is still a vector.

The local derivative is simply a vector of ones: $$ \frac{ \partial Z}{\partial \mathbf{b}} = \frac{\partial}{\partial \mathbf{b}} W \times X + \mathbf{b} = \mathbf{1} $$

That means our complete derivative is a matrix multiplication, that looks as follows (e.g. 2 samples with 3 outputs): $$ \frac{\partial L}{\partial Z} \times \mathbf{1} = \begin{bmatrix} . &. &. \\ . &. &. \end{bmatrix} \begin{bmatrix} 1\\ 1\\ 1\\ \end{bmatrix} $$

Note that this is the sum of the rows.

And that's where db2=np.sum(dz2, axis=0, keepdims=True) comes from. It is simply an abbreviation for the matrix multiplication of the local and the upstream derivatives.

Answer 3

first, you must correct your formula for the gradient of the sigmoid function.

The first derivative of sigmoid function is: (1−σ(x))σ(x)

Your formula for dz2 will become: dz2 = (1-h2)*h2 * dh2

You must use the output of the sigmoid function for σ(x) not the gradient.

You must sum the gradient for the bias as this gradient comes from many single inputs (the number of inputs = batch size). Thus, we must accumulate them to update the biases of layer 2. However, for the gradients come to layer 1, since they come from many nodes of layer 2, you have to sum all the gradient for updating the biases and weights in layer 1. This case is different from the sum of biases in layer 2.

My implement for two fully-connected layers with the activation functions are sigmoid functions:

lr = 1e-3
f = lambda x: 1.0/(1.0 + np.exp(-x))
# pass through layer 1
out_l1 = np.dot(x, W_1) + b_1

out_s1 = f(out_l1)

# pass through layer 2
out_l2 = np.dot(x, W_2) + b_2

out_s2 = f(out_l2)

loss = get_loss(out_s2, y)

# BACKWARD PASS
grad = out_s2 - y

d_h2 = (1 - out_s2) * out_s2 * grad

# Accumulate the gradient come from all examples
d_W2 = out_s1.T.dot(d_h2)
d_b2 = np.sum(d_h2, axis=0, keepdims=True)

# sum of gradient come out from prev node:
grad_1 = np.sum(d_W2.T, axis=0, keepdims=True)
d_h1 = (1 - out_l1) * out_l1 * grad_1

d_W1 = x.T.dot(d_h1)
d_b1 = np.sum(d_h1, axis=0, keepdims=True)

W_1 -= d_W1 * lr
b_1 -= d_b1 * lr

W_2 -= d_W2 * lr
b_2 -= d_b2 * lr

Answer 4

I'm going to elaborate a bit more on oezguensi's answer and see if we can resolve the problems with the dimensions which seem a bit off.

In the post you had 2 samples with 3 outputs (classes). In that case, $\frac{\partial L}{\partial Z}$ should have dimensions (3,2): https://en.wikipedia.org/wiki/Matrix_calculus#Scalar-by-matrix .

On the other hand $\mathbf{b}$ should be a (2,1) matrix of bias vectors, since you broadcast the bias twice, once per sample. Each one of these bias vectors has 3 components corresponding to the biases of each linear combination.

Let's type it out. Assume each data point has $D = 3$ features: So we have: $S = 2$ samples (batch size). $D = 3$ features. $M = 3$ classes (outputs).

We have: $X_{S\times D} = \begin{bmatrix} x_{11} & x_{12} & x_{13} & x_{14}\\ x_{21} & x_{22} & x_{23} & x_{24}\\ \end{bmatrix}$ and: $W_{D\times M} = \begin{bmatrix} w_{11} & w_{12} & w_{13} \\ w_{21} & w_{22} & w_{23} \\ w_{31} & w_{32} & w_{33} \\ w_{41} & w_{42} & w_{43} \\ \end{bmatrix}$

Then we have: $Z_{S\times M} = W \times X = \begin{bmatrix} z_{11} & z_{12} & z_{13} \\ z_{21} & z_{22} & z_{23} \\ \end{bmatrix}$

For each row of the $Z$ matrix we have a bias vector $\overrightarrow{b}_{M \times 1} = (b_1, b_2, b_3)$ when we broadcast we get the bias matrix $\mathbf{b} = \begin{bmatrix} \overrightarrow{b} \\ \overrightarrow{b} \\ \end{bmatrix}$

Let's write the derivatives (using the Wikipedia formula above): $(\partial{L}/\partial{Z})_{M \times S} = \begin{bmatrix} \partial{L}/\partial{z_{11}} & \partial{L}/\partial{z_{21}} \\ \partial{L}/\partial{z_{12}} & \partial{L}/\partial{z_{22}} \\ \partial{L}/\partial{z_{13}} & \partial{L}/\partial{z_{23}} \\ \end{bmatrix}$

The bias: $(\partial{Z}/\partial\overrightarrow{b})_{S \times 1} = \begin{bmatrix} 1 \\ 1 \\ \end{bmatrix}$

The derivative of the upstream with respect to the bias vector: $$\frac{\partial L}{\partial \overrightarrow{b}} = \frac{\partial L}{\partial Z}\frac{\partial Z}{\partial \overrightarrow{b}}$$

Has shape $M\times 1$ and is the sum along the columns of the $(\partial{L}/\partial{Z})_{M \times S}$ matrix. Each entry of this matrix gives you the downstream gradient of the entries of $\overrightarrow{b}$.

But it's important to note that it is common to give the upstream derivative matrix as its transpose, with shape $S \times M$, that is: batch size as rows and classes as columns. In this case, you sum along the rows of the transpose.

So just keep an eye on the shape of the upstream gradient to find out which direction to sum.