I am working on a binary classification problem using CNN model, the model designed using tensorflow framework, in most GitHub projects that I saw, they use "softmax cross entropy with logits" v1 and v2 as loss function, my questions are:
1- Why this loss method is the most used one?
2- What is the type of this function because of the name I am confused about its type.
3- What is the equation for the function? in tensorflow website, the equation is not available.
I think there is some confusion here. Softmax is usually an activation function which you will use in your output layer, and cross-entropy is the loss function that you will use.
This activation function outputs the probability for each class, it is defined as
$\sigma(\bf{y_i}) = \frac{e^{\bf{y_i}}}{\sum_{j}{e^{\bf{y_j}}}}$.
For example, in a problem with 2 class labels. Then if we have some outputs from our neural network like $y$=[3, 4] then we can get the probability of each output classes by using the softmax function on these outputs.
import numpy as np
x = [3, 4]
np.exp(x)/np.sum(np.exp(x))
[0.26894142, 0.73105858]
You will see that these are probabilities and do sum to 1.
Then we can get the class of the input by seeing which probability is higher. In this case class 2 has a probability of 73.11%, so the predicted class label, $\hat{y} = 1$.
Cross-entropy is a loss function that is used for classification tasks. For binary classification it is defined as
$H(p, q) = -y\log(p) - (1-y)\log(1-p)$.
Let's assume that the real class of the above example is 0, $y=0$. Then we made a mistake and you can see that
$H(p, q) = -0\log(0.26894142) - (1-0)\log(1-0.26894142) = 0.313$.
That is the loss that is used for backpropagation.
