I have trouble implementing back propogation for multi class classification

My neural network has 2 layers

Forward propagation

X -> L1 -> L2

weights W are initialized as random

np.random.randn(this_layer_units, previous_layer_units) * 0.01

X is input of size (no_features * number of examples)

Z1 = (w1 * x) + b1

A1 = relu(Z1)

L1 has ReLu activation

Z2 = (w2 * A1) + b2

A2 = softmax(Z1)

L2 has softmax activation

cost is caluclated using cross entropy

cost = -(1/m)*np.sum((Y * np.log(A2) ) + ((1 - Y)*np.log(1-A2)))

Back propagation

derivative of cost is calculated

dA2 = -(1/m)*(np.divide(Y, A2) - np.divide(1 - Y, 1 - A2))

dA2 = derivative of cost w.r.t softmax

Y = one hot encoded True values

softmax is

np.exp(z)/ np.sum(np.exp(z))

now, how do i find dZ2 (derivative of Z2) w.r.t dA2 (softmax activation)

Link to entire jupyter notebook code

  • $\begingroup$ Backprop is never Specific to a dataset.. Here's mine on MNIST github.com/AdityaSoni19031997/Machine-Learning/blob/master/… $\endgroup$ – Aditya Sep 23 '18 at 17:44
  • $\begingroup$ It won't be softmax of Z1 though.. Also your derivative calculation isn't correct imho.. Here we ain't doing normal Scaler derivatives... Everything is Vector, so things change a bit... Try reading blogs to quench your Knowledge base... $\endgroup$ – Aditya Sep 23 '18 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.