What's the proper way to do back propagation in Deep Fully Connected Neural Network for binary classification

Question

I tried to implement a Deep fully connected neural network for binary classification using python and numpy and used Gradient Descent as optimization algorithm.

Turns out my model is heavily under fitting, even after 1000 epochs. My loss never improves beyond 0.69321, i tried checking my weight derivatives and instantly realized they're very small ( as small as 1e-7 ), such small gradients are causing my model to never have bigger gradient descent updates and never reach the global minima. I will detail out the math/pseudo code for forward and backward propagation's, please let me know if I'm on the right track. I will follow the naming convention used in DeepLearning.ai By Andrew Ng.

Say we have 4 layer neural network with only one node at the output layer to classify between 0/1.

X -> Z1 - > A1 - > Z2 - > A2 - > Z3 - > A3 - > Z4 - > A4

Forward propagation

Z1 = W1 dot_product X + B1
A1 = tanh_activation(Z1)

Z2 = W2 dot_product A1 + B2
A2 = tanh_activation(Z2)

Z3 = W3 dot_product A2 + B3
A3 = tanh_activation(Z3)

Z4 = W4 dot_product A3 + B4
A4 = sigmoid_activation(Z4)

Backward Propagation

DA4 = -( Y / A4 + (1 - Y /  1 - A4 ) ) ( derivative of output activations or logits w.r.t to loss function )

DZ4 = DA4 * derivative_tanh(Z4) ( derivative of tanh activation, which I assume is ( 1 - (Z4 ) ^ 2 ) )
Dw4 = ( dZ4 dot_produt A3.T ) / total_number_of_samples
Db4 = np.sum(DZ4, axis = 1, keepdims = True ... ) / total_number_of_samples
DA3 = W4.T dot_product(DZ4)


DZ3 = DA3 * derivative_tanh( Z3 )
DW3 = ( DZ3 dot_product A2.T ) / total_number_of_samples
DB3 = np.sum( DZ3, .. ) / total_number_of_samples
DA2 = W3.T dot_product(DZ3)


DZ2 = DA2 * derivative_tanh( Z2 )
DW2 = ( DZ2 dot_product A1.T ) / total_number_of_samples
DB2 = np.sum( DZ2, .. ) / total_number_of_samples
DA1 = W2.T dot_product(DZ2)



DZ1 = DA1 * derivative_tanh( Z1 )
DW1 = ( DZ1 dot_product X.T ) / total_number_of_samples
DB1 = np.sum( DZ1, .. ) / total_number_of_samples

After the above back propagation steps I updated the weights and biases using gradient descent with their respective derivatives. But, no matter how many times I run the algorithm, the model never improves it's loss beyond 0.69 and the derivatives of output weights ( in my case dW4 ) is pretty low 1e-7. I'm assuming that either my derivative_tanh function or my calculations of dZ is really off, which is causing bad loss values to propagate back to the network. Please share your thoughts whether my implementation of backprop is valid or not. TIA. I went through back propagation gradient descent calculus

and

how to optimize weights of neural network .. and many other blogs, but couldn't find for what I was looking for.

Answer 1

The problem faced with back propagation with activation functions that has a max limit used in a Deep Neural Network (i.e. more then 2 layer network) is that the deeper the network is the quicker your back propagation will degrade.

Take for example your tanh activation function

If say you backpropagate from the output layer to the hidden layer and your tanh derivative is 0.25 then the next layer will only be limited by 0.25 from your tanh function your derived. Since you compute the next layer's derivative which in this case can be 0.5 of the total input and the previous layer's derivative totals to 0.25 then your derivative would be a total of 0.125. By the time the back propagation reaches the input layer, the weights would change by such a small fraction that it does not even matter.

This is why the ReLu function was such a big breakthrough as it does not have a max limit. There should be methods to counter this effect but ReLu is way more effective.

Answer 2

@SandMan Thank you for all the suggestions, I did check the Loss, DZ4 and DW4 values before posting this question, as you suspected the problem start's much earlier i.e during my calculation of DZ4, my implementation sigmoid_derivative seemed to be not quite to the point, which caused very low values to propagate back through the network and my gradient descent was never able to take bigger steps towards global minima. I was able to fix it and achieve 100% accuracy with 0.2% loss, I was about to post an answer with my corrections. I will detail out the pseudo code here:

Forward Propagation

Z1 = W1 dot_product X + B1
A1 = relu_activation(Z1)

Z2 = W2 dot_product A1 + B2
A2 = relu_activation(Z2)

Z3 = W3 dot_product A2 + B3
A3 = relu_activation(Z3)

Z4 = W4 dot_product A3 + B4
A4 = sigmoid_activation(Z4)

Backward Propagation

DA4 = -( Y / A4 + (1 - Y /  1 - A4 ) ) ( derivative of output activations or logits w.r.t to loss function )

DZ4 = DA4 * sigmoid_derivative(Z4) ( derivative of sigmoid activation, which I missed initially )
Dw4 = ( dZ4 dot_produt A3.T ) / total_number_of_samples
Db4 = np.sum(DZ4, axis = 1, keepdims = True ... ) / total_number_of_samples
DA3 = W4.T dot_product(DZ4)


DZ3 = derivative_relu(DA3, Z3 )
DW3 = ( DZ3 dot_product A2.T ) / total_number_of_samples
DB3 = np.sum( DZ3, .. ) / total_number_of_samples
DA2 = W3.T dot_product(DZ3)


DZ2 = derivative_relu( DA2, Z2 )
DW2 = ( DZ2 dot_product A1.T ) / total_number_of_samples
DB2 = np.sum( DZ2, .. ) / total_number_of_samples
DA1 = W2.T dot_product(DZ2)



DZ1 = derivative_relu(DA1, Z1 )
DW1 = ( DZ1 dot_product X.T ) / total_number_of_samples
DB1 = np.sum( DZ1, .. ) / total_number_of_samples

My Sigmoid derivative before

def sigmoid_derivative(x):
 return x * ( 1 - x )

My Sigmoid derivative after

def simoid_derivative(x):
 return np.multiply(sigmoid(x), (1 - sigmoid(x) ) 

Thanks once again for your comments.