1
$\begingroup$

For binary classification irrespective of the model used, the sigmoid function is a good choice for output layer because the actual output value ‘Y’ is either 0 or 1 so it makes sense for predicted output value to be a number between 0 and 1.

My confusion is that is there a binary step function in the output layer which squashes the values of the linear combination of weights and and inputs to 0 or 1? Does classification means always applying a thresholding function on top of a linear or non-linear function which is in the hidden layer?

Say the predicted output value is 0.75 and actual Y is 0. Then, how is 0.75 converted to 1? The loss function would calculate the error as actual - predicted = 0-0.75 = -0.75

Can somebody please explain the math or point out some links where the working steps are shown? Thank you.

$\endgroup$
2
$\begingroup$

I think following picture by Rubens Zimbres can express how this process will be done except loss function part: img

Is there a binary step function in the output layer which squashes the values of the linear combination of weights and and inputs to 0 or 1?

Answer : Yes if you consider following example shows how Sigmoid squishes values to be between 0 and 1: img

Does classification means always applying a thresholding function on top of a linear or non-linear function which is in the hidden layer?

Answer: It does.

Say the predicted output value is 0.75 and actual Y is 0. Then, how is 0.75 converted to 1? The loss function would calculate the error as actual - predicted = 0-0.75 = -0.75

Answer: Loss function and calculation of error for classification-based NN is not just by computation of $Y_Actual - Y_prediction$

but It is used normally by combination of Softmax activation function and cross-entropy as follow: img

See the whole training process Multinomial Logistic Classification $D(S(wx+b),L)$ img

$\endgroup$
  • $\begingroup$ Awesome! thank you for the illustration and point wise explanation. $\endgroup$ – Sm1 Jun 18 '19 at 0:54
  • $\begingroup$ One last question -- why do we need sigmoid and not just a binary threshold function for the last output layer in classification to 0 and 1? $\endgroup$ – Sm1 Jun 18 '19 at 2:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.