For a simple problem of classification (C classes) using the softmax classifier, most people use the cross-entropy loss function to quantify the objective. The cross-entropy loss is:
$$L = -\sum_{i=1}^C y_ilog(p_i)$$
where $p_i$ is the classifier prediction score and $y_i$ is the ground-truth labels. To update the parameters, gradient descent is used to make the distribution of $p_i$ as similar as possible to the distribution of $y_i$. Let's say for an arbitrary point in space ($x_i$) we have the ground truth label $y_i = [1, 0, 0]$ which means for a $C=3$ class problem, $x_i$ belongs to the class 0. And let's say the softmax classifier has a prediction of $p_i = [0.3, 0.6, 0.1]$ in the first iteration. When optimizing $L$ with gradient descent, the prediction changes like this:
- [iteration 0] ....... $p_i = [0.3, 0.6, 0.1]$
- [iteration 1] ....... $p_i = [0.4, 0.5, 0.08]$
- [iteration 2] ....... $p_i = [0.55, 0.3, 0.02]$
- ...........................................................
- [iteration n] ....... $p_i = [0.98, 0.001, 0.0001]$
So it seems like gradient descent is trying to increase the value for index 0 and decrease value for the other indices. Based on my understanding, cross-entropy is only evaluated on the correct class, so the other terms will be zeroed out because of the ground truth array $y_i = [1, 0, 0]$:
$L = -\sum_{i=1}^C y_ilog(p_i) = - 1 * log(0.3) = 1.2$
I don't understand the mechanism here. Is it the gradient descent doing these modifications or is it the cross-entropy loss we define or is it the combination of both? How does the prediction score for the correct class increases and the prediction score for the incorrect classes decrease while optimizing?