I was trying to implement neural network from scratch to understand the maths behind it. My problem is completely related to backpropagation when we take derivative with respect to bias) and I derived all the equations used in backpropagation. Now every equation is matching with the code for neural network except for that the derivative with respect to biases.



#back prop


I looked up online for the code, and i want to know why do we add up the matrix and then the scalar db2=np.sum(dz2,axis=0,keepdims=True) is subtracted from the original bias, why not the matrix as a whole is subtracted. Can anyone help me to give some intuion behind it. If i take partial derivative of loss with respect to bias it will give me upper gradient only which is dz2 because z2=h1.dot(theta2)+b2 h1 and theta will be 0 and b2 will be 1. So the upper term will be left.


4 Answers 4


The bias term is very simple, which is why you often don't see it calculated. In fact

db2 = dz2

So your update rules for bias on a single item are:

b2 += -alpha * dz2


b1 += -alpha * dz1

In terms of the maths, if your loss is $J$, and you know $\frac{\partial J}{\partial z_i}$ for a given neuron $i$ which has bias term $b_i$ . . .

$$\frac{\partial J}{\partial b_i} = \frac{\partial J}{\partial z_i} \frac{\partial z_i}{\partial b_i}$$


$$\frac{\partial z_i}{\partial b_i} = 1$$

because $z_i = (\text{something unaffected by } b_i) + b_i$

It looks like the code you copied uses the form


because the network is designed to process examples in (mini-)batches, and you therefore have gradients calculated for more than one example at a time. The sum is squashing the results down to a single update. This would be easier to confirm if you also showed update code for weights.

  • $\begingroup$ yes,exactly that is what my thoughts were,because thats how mathematically it looks like.but the code i saw they summed up the matrix and then added it to the b1. $\endgroup$
    – user34042
    Commented Jul 3, 2017 at 19:30
  • $\begingroup$ theta1=theta1-alpha*dw1 theta2=theta2-alpha*dw2 i still don't get it.that way same term will be added to all the different terms in the 'b' vector which otherwise would have had different weights for every single terms.that would make significant difference for neural network to achieve minima. $\endgroup$
    – user34042
    Commented Jul 3, 2017 at 19:30
  • $\begingroup$ @user34042: Something doesn't seem right to me - could you link the source you got that code from? I wonder if the source got it wrong because it has mixed and matched mini-batch code with simple online gradient descent. $\endgroup$ Commented Jul 3, 2017 at 19:35
  • $\begingroup$ wildml.com/2015/09/implementing-a-neural-network-from-scratch here it is. $\endgroup$
    – user34042
    Commented Jul 3, 2017 at 19:36
  • $\begingroup$ I think the source has it wrong. The NN will still kind of work with all bias values the same, so they may not have noticed. And as I mentioned, you might actually use that code in a batch-based scenario, so it could just be a cut&paste error. $\endgroup$ Commented Jul 3, 2017 at 19:39

I would like to explain the meaning of db2=np.sum(dz2,axis=0,keepdims=True) as it also confused me once and it didn't get answered.

The derivative of L (loss) w.r.t. b is the upstream derivative multiplied with the local derivate: $$ \frac{ \partial L}{\partial \mathbf{b}} = \frac{ \partial L}{\partial Z} \frac{ \partial Z}{\partial \mathbf{b}} $$

If we have multiple samples Z and L are both matrices. b is still a vector.

The local derivative is simply a vector of ones: $$ \frac{ \partial Z}{\partial \mathbf{b}} = \frac{\partial}{\partial \mathbf{b}} W \times X + \mathbf{b} = \mathbf{1} $$

That means our complete derivative is a matrix multiplication, that looks as follows (e.g. 2 samples with 3 outputs): $$ \frac{\partial L}{\partial Z} \times \mathbf{1} = \begin{bmatrix} . &. &. \\ . &. &. \end{bmatrix} \begin{bmatrix} 1\\ 1\\ 1\\ \end{bmatrix} $$

Note that this is the sum of the rows.

And that's where db2=np.sum(dz2, axis=0, keepdims=True) comes from. It is simply an abbreviation for the matrix multiplication of the local and the upstream derivatives.

  • 2
    $\begingroup$ This answer does not seem to be correct as pointed out already. axis=0 means we're summing the columns, so the result will be a 1x3 matrix. $\endgroup$
    – Snowball
    Commented Oct 22, 2020 at 22:37
  • $\begingroup$ I think it should be (1,1,1) * dL/dz. The ones need to be shaped (1,3) and not (3,1) $\endgroup$
    – deekay42
    Commented Jan 29, 2021 at 3:29

first, you must correct your formula for the gradient of the sigmoid function.

The first derivative of sigmoid function is: (1−σ(x))σ(x)

Your formula for dz2 will become: dz2 = (1-h2)*h2 * dh2

You must use the output of the sigmoid function for σ(x) not the gradient.

You must sum the gradient for the bias as this gradient comes from many single inputs (the number of inputs = batch size). Thus, we must accumulate them to update the biases of layer 2. However, for the gradients come to layer 1, since they come from many nodes of layer 2, you have to sum all the gradient for updating the biases and weights in layer 1. This case is different from the sum of biases in layer 2.

My implement for two fully-connected layers with the activation functions are sigmoid functions:

lr = 1e-3
f = lambda x: 1.0/(1.0 + np.exp(-x))
# pass through layer 1
out_l1 = np.dot(x, W_1) + b_1

out_s1 = f(out_l1)

# pass through layer 2
out_l2 = np.dot(x, W_2) + b_2

out_s2 = f(out_l2)

loss = get_loss(out_s2, y)

grad = out_s2 - y

d_h2 = (1 - out_s2) * out_s2 * grad

# Accumulate the gradient come from all examples
d_W2 = out_s1.T.dot(d_h2)
d_b2 = np.sum(d_h2, axis=0, keepdims=True)

# sum of gradient come out from prev node:
grad_1 = np.sum(d_W2.T, axis=0, keepdims=True)
d_h1 = (1 - out_l1) * out_l1 * grad_1

d_W1 = x.T.dot(d_h1)
d_b1 = np.sum(d_h1, axis=0, keepdims=True)

W_1 -= d_W1 * lr
b_1 -= d_b1 * lr

W_2 -= d_W2 * lr
b_2 -= d_b2 * lr

I'm going to elaborate a bit more on oezguensi's answer and see if we can resolve the problems with the dimensions which seem a bit off.

In the post you had 2 samples with 3 outputs (classes). In that case, $\frac{\partial L}{\partial Z}$ should have dimensions (3,2): https://en.wikipedia.org/wiki/Matrix_calculus#Scalar-by-matrix .

On the other hand $\mathbf{b}$ should be a (2,1) matrix of bias vectors, since you broadcast the bias twice, once per sample. Each one of these bias vectors has 3 components corresponding to the biases of each linear combination.

Let's type it out. Assume each data point has $D = 3$ features: So we have: $S = 2$ samples (batch size). $D = 3$ features. $M = 3$ classes (outputs).

We have: $X_{S\times D} = \begin{bmatrix} x_{11} & x_{12} & x_{13} & x_{14}\\ x_{21} & x_{22} & x_{23} & x_{24}\\ \end{bmatrix}$ and: $W_{D\times M} = \begin{bmatrix} w_{11} & w_{12} & w_{13} \\ w_{21} & w_{22} & w_{23} \\ w_{31} & w_{32} & w_{33} \\ w_{41} & w_{42} & w_{43} \\ \end{bmatrix}$

Then we have: $Z_{S\times M} = W \times X = \begin{bmatrix} z_{11} & z_{12} & z_{13} \\ z_{21} & z_{22} & z_{23} \\ \end{bmatrix}$

For each row of the $Z$ matrix we have a bias vector $\overrightarrow{b}_{M \times 1} = (b_1, b_2, b_3)$ when we broadcast we get the bias matrix $\mathbf{b} = \begin{bmatrix} \overrightarrow{b} \\ \overrightarrow{b} \\ \end{bmatrix}$

Let's write the derivatives (using the Wikipedia formula above): $(\partial{L}/\partial{Z})_{M \times S} = \begin{bmatrix} \partial{L}/\partial{z_{11}} & \partial{L}/\partial{z_{21}} \\ \partial{L}/\partial{z_{12}} & \partial{L}/\partial{z_{22}} \\ \partial{L}/\partial{z_{13}} & \partial{L}/\partial{z_{23}} \\ \end{bmatrix}$

The bias: $(\partial{Z}/\partial\overrightarrow{b})_{S \times 1} = \begin{bmatrix} 1 \\ 1 \\ \end{bmatrix}$

The derivative of the upstream with respect to the bias vector: $$\frac{\partial L}{\partial \overrightarrow{b}} = \frac{\partial L}{\partial Z}\frac{\partial Z}{\partial \overrightarrow{b}}$$

Has shape $M\times 1$ and is the sum along the columns of the $(\partial{L}/\partial{Z})_{M \times S}$ matrix. Each entry of this matrix gives you the downstream gradient of the entries of $\overrightarrow{b}$.

But it's important to note that it is common to give the upstream derivative matrix as its transpose, with shape $S \times M$, that is: batch size as rows and classes as columns. In this case, you sum along the rows of the transpose.

So just keep an eye on the shape of the upstream gradient to find out which direction to sum.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.