I was trying to implement neural network from scratch to understand the maths behind it. My problem is completely related to backpropagation when we take derivative with respect to bias) and I derived all the equations used in backpropagation. Now every equation is matching with the code for neural network except for that the derivative with respect to biases.
z1=x.dot(theta1)+b1 h1=1/(1+np.exp(-z1)) z2=h1.dot(theta2)+b2 h2=1/(1+np.exp(-z2)) dh2=h2-y #back prop dz2=dh2*(1-dh2) H1=np.transpose(h1) dw2=np.dot(H1,dz2) db2=np.sum(dz2,axis=0,keepdims=True)
I looked up online for the code, and i want to know
why do we add up the matrix and then the scalar
db2=np.sum(dz2,axis=0,keepdims=True) is subtracted from the original bias, why not the matrix as a whole is subtracted. Can anyone help me to give some intuion behind it. If i take partial derivative of loss with respect to bias it will give me upper gradient only which is dz2 because
z2=h1.dot(theta2)+b2 h1 and theta will be 0 and b2 will be 1. So the upper term will be left.