# What's the proper way to do back propagation in Deep Fully Connected Neural Network for binary classification

I tried to implement a Deep fully connected neural network for binary classification using python and numpy and used Gradient Descent as optimization algorithm.

Turns out my model is heavily under fitting, even after 1000 epochs. My loss never improves beyond 0.69321, i tried checking my weight derivatives and instantly realized they're very small ( as small as 1e-7 ), such small gradients are causing my model to never have bigger gradient descent updates and never reach the global minima. I will detail out the math/pseudo code for forward and backward propagation's, please let me know if I'm on the right track. I will follow the naming convention used in DeepLearning.ai By Andrew Ng.

Say we have 4 layer neural network with only one node at the output layer to classify between 0/1.

X -> Z1 - > A1 - > Z2 - > A2 - > Z3 - > A3 - > Z4 - > A4

Forward propagation

Z1 = W1 dot_product X + B1
A1 = tanh_activation(Z1)

Z2 = W2 dot_product A1 + B2
A2 = tanh_activation(Z2)

Z3 = W3 dot_product A2 + B3
A3 = tanh_activation(Z3)

Z4 = W4 dot_product A3 + B4
A4 = sigmoid_activation(Z4)


Backward Propagation

DA4 = -( Y / A4 + (1 - Y /  1 - A4 ) ) ( derivative of output activations or logits w.r.t to loss function )

DZ4 = DA4 * derivative_tanh(Z4) ( derivative of tanh activation, which I assume is ( 1 - (Z4 ) ^ 2 ) )
Dw4 = ( dZ4 dot_produt A3.T ) / total_number_of_samples
Db4 = np.sum(DZ4, axis = 1, keepdims = True ... ) / total_number_of_samples
DA3 = W4.T dot_product(DZ4)

DZ3 = DA3 * derivative_tanh( Z3 )
DW3 = ( DZ3 dot_product A2.T ) / total_number_of_samples
DB3 = np.sum( DZ3, .. ) / total_number_of_samples
DA2 = W3.T dot_product(DZ3)

DZ2 = DA2 * derivative_tanh( Z2 )
DW2 = ( DZ2 dot_product A1.T ) / total_number_of_samples
DB2 = np.sum( DZ2, .. ) / total_number_of_samples
DA1 = W2.T dot_product(DZ2)

DZ1 = DA1 * derivative_tanh( Z1 )
DW1 = ( DZ1 dot_product X.T ) / total_number_of_samples
DB1 = np.sum( DZ1, .. ) / total_number_of_samples



After the above back propagation steps I updated the weights and biases using gradient descent with their respective derivatives. But, no matter how many times I run the algorithm, the model never improves it's loss beyond 0.69 and the derivatives of output weights ( in my case dW4 ) is pretty low 1e-7. I'm assuming that either my derivative_tanh function or my calculations of dZ is really off, which is causing bad loss values to propagate back to the network. Please share your thoughts whether my implementation of backprop is valid or not. TIA. I went through back propagation gradient descent calculus

and

how to optimize weights of neural network .. and many other blogs, but couldn't find for what I was looking for.

The problem faced with back propagation with activation functions that has a max limit used in a Deep Neural Network (i.e. more then 2 layer network) is that the deeper the network is the quicker your back propagation will degrade.

Take for example your tanh activation function

If say you backpropagate from the output layer to the hidden layer and your tanh derivative is 0.25 then the next layer will only be limited by 0.25 from your tanh function your derived. Since you compute the next layer's derivative which in this case can be 0.5 of the total input and the previous layer's derivative totals to 0.25 then your derivative would be a total of 0.125. By the time the back propagation reaches the input layer, the weights would change by such a small fraction that it does not even matter.

This is why the ReLu function was such a big breakthrough as it does not have a max limit. There should be methods to counter this effect but ReLu is way more effective.

• thanks for the response. I agree that ReLu is more stable than tanh and deeper the network, that much faster the gradient descent degrades. Your explanation makes sense, but I was looking if my implementation of back propagation was even right in the first place. I was looking for some suggestions if my Math is right in the above pseudo code. I realized that on my DZ4 calculation I had to take derivative w.r.t to sigmoid activation but not tanh, as it belongs to output layer. With that change my network is working just about fine, definitely some more room for improvement. Sep 18 '19 at 16:16
• like DZ4 = DA4 * sigmoid_derivative(Z4)  ; changing only this line and using tanh gave me better results than before, but as you said my loss didn't improve because of very small gradient descent steps. I also tried changing my activation to ReLu and changed my backpropagation calculations like, DZ[L] = DA[L] * relu_derivative(Z[L]) , and kept my DW[L]  calculations consistent with how I did before ... but even that doesn't seem to improve my gradients a great deal and in turn never takes bigger steps towards local minima. Can't identify where my calculations are going wrong Sep 18 '19 at 16:42
• Hi there, I decided to revisit your question. I assume you meant to get the derivative of the sigmoid activision function for DZ4 in your question as you fixed it in a comment here. It seems your maths is on point. The only problem I can seem to find is that you might not have kept in mind the dot_product of the derivatives for the weight w.r.t the loss function from previous layers as your backpropagation progresses but as you stated your DW4 is small. Sep 19 '19 at 16:00
• This indicates to me that the problem occurs much earlier. I would say step through your code and see what the values are for your Loss, DA4, DZ4, and DW4. You should quickly see where a calculation anomaly occurs if the loss function is large value. Other than keeping in mind the derivatives of previous layers, looking from an abstract point of view it seems you understand what needs to occur to get your neural network to train Sep 19 '19 at 16:00
• Lastly, this article really explained backpropagation and the calculus behind it very well. The fine details are mentioned and if you take the time to read carefully you should be able to derive backpropagation equations with any functions thrown your way. medium.com/binaryandmore/… Sep 19 '19 at 16:06

@SandMan Thank you for all the suggestions, I did check the Loss, DZ4 and DW4 values before posting this question, as you suspected the problem start's much earlier i.e during my calculation of DZ4, my implementation sigmoid_derivative seemed to be not quite to the point, which caused very low values to propagate back through the network and my gradient descent was never able to take bigger steps towards global minima. I was able to fix it and achieve 100% accuracy with 0.2% loss, I was about to post an answer with my corrections. I will detail out the pseudo code here:

Forward Propagation

Z1 = W1 dot_product X + B1
A1 = relu_activation(Z1)

Z2 = W2 dot_product A1 + B2
A2 = relu_activation(Z2)

Z3 = W3 dot_product A2 + B3
A3 = relu_activation(Z3)

Z4 = W4 dot_product A3 + B4
A4 = sigmoid_activation(Z4)



Backward Propagation

DA4 = -( Y / A4 + (1 - Y /  1 - A4 ) ) ( derivative of output activations or logits w.r.t to loss function )

DZ4 = DA4 * sigmoid_derivative(Z4) ( derivative of sigmoid activation, which I missed initially )
Dw4 = ( dZ4 dot_produt A3.T ) / total_number_of_samples
Db4 = np.sum(DZ4, axis = 1, keepdims = True ... ) / total_number_of_samples
DA3 = W4.T dot_product(DZ4)

DZ3 = derivative_relu(DA3, Z3 )
DW3 = ( DZ3 dot_product A2.T ) / total_number_of_samples
DB3 = np.sum( DZ3, .. ) / total_number_of_samples
DA2 = W3.T dot_product(DZ3)

DZ2 = derivative_relu( DA2, Z2 )
DW2 = ( DZ2 dot_product A1.T ) / total_number_of_samples
DB2 = np.sum( DZ2, .. ) / total_number_of_samples
DA1 = W2.T dot_product(DZ2)

DZ1 = derivative_relu(DA1, Z1 )
DW1 = ( DZ1 dot_product X.T ) / total_number_of_samples
DB1 = np.sum( DZ1, .. ) / total_number_of_samples



My Sigmoid derivative before

def sigmoid_derivative(x):
return x * ( 1 - x )


My Sigmoid derivative after

def simoid_derivative(x):
return np.multiply(sigmoid(x), (1 - sigmoid(x) )