I am developing a digit classifer with MNIST Dataset. I have read that for classification problems softmax activation function is used, as it maps last layer neuron's outputs into probabilities.

While backpropagating we need derivative of the activation function say f'(x) to calculate weights adjustment (delta_weights)

ΔW = -η(gradient of Error)

ΔW = η( desired_output - output )f'(net_value)X

Now what i am doing is after calculating dot product of second last layer's output and last layer weights and now i have net_value of the last layer and this is where i am stucked. should i ?

(1) Pass net_value to softmax directly i.e. softmax(net_value)


(2) First squash the net_value by logistic(sigmoid) activation and then pass output of that in softmax. i.e. softmax( sigmoid(net_value) )

After that what derivative function f'(x) should i use for backpropagating last layer?

I have tried using derivative of sigmoid for backpropagating last layer ( layer with softmax activation with net_value directly passing to the softmax ) and it surprisingly gave me accuracy of ~90% in MNIST's 10000 test images


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.