5
$\begingroup$

I am working on a binary classification problem using CNN model, the model designed using tensorflow framework, in most GitHub projects that I saw, they use "softmax cross entropy with logits" v1 and v2 as loss function, my questions are:

1- Why this loss method is the most used one?

2- What is the type of this function because of the name I am confused about its type.

3- What is the equation for the function? in tensorflow website, the equation is not available.

$\endgroup$
4
$\begingroup$

I think there is some confusion here. Softmax is usually an activation function which you will use in your output layer, and cross-entropy is the loss function that you will use.

Softmax

This activation function outputs the probability for each class, it is defined as

$\sigma(\bf{y_i}) = \frac{e^{\bf{y_i}}}{\sum_{j}{e^{\bf{y_j}}}}$.

For example, in a problem with 2 class labels. Then if we have some outputs from our neural network like $y$=[3, 4] then we can get the probability of each output classes by using the softmax function on these outputs.

import numpy as np

x = [3, 4]
np.exp(x)/np.sum(np.exp(x))

[0.26894142, 0.73105858]

You will see that these are probabilities and do sum to 1.

Then we can get the class of the input by seeing which probability is higher. In this case class 2 has a probability of 73.11%, so the predicted class label, $\hat{y} = 1$.

Cross-entropy

Cross-entropy is a loss function that is used for classification tasks. For binary classification it is defined as

$H(p, q) = -y\log(p) - (1-y)\log(1-p)$.

Let's assume that the real class of the above example is 0, $y=0$. Then we made a mistake and you can see that

$H(p, q) = -0\log(0.26894142) - (1-0)\log(1-0.26894142) = 0.313$.

That is the loss that is used for backpropagation.

$\endgroup$
1
  • $\begingroup$ You should use 'tf.nn.sigmoid_cross_entropy_with_logits' with one hot encoding. $\endgroup$ – Aaditya ura Mar 14 '20 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.