1
$\begingroup$

I read this and have an ambiguity.

I try to understand well how to calculate the derivative of Loss w.r.t to bias.

In this question, we have this definition:

np.sum(dz2,axis=0,keepdims=True)

Then in Casper's comment, he said that the The derivative of L (loss) w.r.t. b is the sum of the rows

$$ \frac{\partial L}{\partial Z} \times \mathbf{1} = \begin{bmatrix} . &. &. \\ . &. &. \end{bmatrix} \begin{bmatrix} 1\\ 1\\ 1\\ \end{bmatrix} $$

But actually, using axis=0, is it not the sum of the columns of ∂𝐿/∂𝑍 ?

I saw another examples and it seems that they do the sum per column. I don't get how to get this result. Could you give the details with a matrix example?

$\endgroup$
1
  • 1
    $\begingroup$ Axis = 0 sums along the rows - axis = 0 operation along the rows, axis = 1 along the columns, axis = 2 along the depth and so forth. $\endgroup$
    – bonfab
    Commented Jun 2, 2020 at 6:30

1 Answer 1

1
$\begingroup$

The dimensions seem to be a bit off. In the post you had 2 samples with 3 outputs (classes). In that case, $\frac{\partial L}{\partial Z}$ should have dimensions (3,2): https://en.wikipedia.org/wiki/Matrix_calculus#Scalar-by-matrix .

On the other hand $\mathbf{b}$ should be a (2,1) matrix of bias vectors, since you broadcast the bias twice, once per sample. Each one of these bias vectors has 3 components corresponding to the biases of each linear combination.

Let's type it out. Assume each data point has $D = 3$ features: So we have: $S = 2$ samples (batch size). $D = 3$ features. $M = 3$ classes (outputs).

We have: $X_{S\times D} = \begin{bmatrix} x_{11} & x_{12} & x_{13} & x_{14}\\ x_{21} & x_{22} & x_{23} & x_{24}\\ \end{bmatrix}$ and: $W_{D\times M} = \begin{bmatrix} w_{11} & w_{12} & w_{13} \\ w_{21} & w_{22} & w_{23} \\ w_{31} & w_{32} & w_{33} \\ w_{41} & w_{42} & w_{43} \\ \end{bmatrix}$

Then we have: $Z_{S\times M} = W \times X = \begin{bmatrix} z_{11} & z_{12} & z_{13} \\ z_{21} & z_{22} & z_{23} \\ \end{bmatrix}$

For each row of the $Z$ matrix we have a bias vector $\overrightarrow{b}_{M \times 1} = (b_1, b_2, b_3)^T$ when we broadcast we get the bias matrix $\mathbf{b} = \begin{bmatrix} \overrightarrow{b} \\ \overrightarrow{b} \\ \end{bmatrix}$

Let's write the derivatives (using the Wikipedia formula above): $(\partial{L}/\partial{Z})_{M \times S} = \begin{bmatrix} \partial{L}/\partial{z_{11}} & \partial{L}/\partial{z_{21}} \\ \partial{L}/\partial{z_{12}} & \partial{L}/\partial{z_{22}} \\ \partial{L}/\partial{z_{13}} & \partial{L}/\partial{z_{23}} \\ \end{bmatrix}$

The bias: $(\partial{Z}/\partial\overrightarrow{b})_{S \times 1} = \begin{bmatrix} 1 \\ 1 \\ \end{bmatrix}$

The derivative of the upstream with respect to the bias vector: $$\frac{\partial L}{\partial \overrightarrow{b}} = \frac{\partial L}{\partial Z}\frac{\partial Z}{\partial \overrightarrow{b}}$$

Has shape $M\times 1$ and is the sum along the columns of the $(\partial{L}/\partial{Z})_{M \times S}$ matrix. Each entry of this matrix gives you the downstream gradient of the entries of $\overrightarrow{b}$.

But it's important to note that it is common to give the upstream derivative matrix as its transpose, with shape $S \times M$, that is: batch size as rows and classes as columns. In this case, you sum along the rows of the transpose.

So just keep an eye on the shape of the upstream gradient to find out which direction to sum.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.