How the combination of cross entropy loss and gradient descent penalizes and rewards

Question

For a simple problem of classification (C classes) using the softmax classifier, most people use the cross-entropy loss function to quantify the objective. The cross-entropy loss is:

$$L = -\sum_{i=1}^C y_ilog(p_i)$$

where $p_i$ is the classifier prediction score and $y_i$ is the ground-truth labels. To update the parameters, gradient descent is used to make the distribution of $p_i$ as similar as possible to the distribution of $y_i$. Let's say for an arbitrary point in space ($x_i$) we have the ground truth label $y_i = [1, 0, 0]$ which means for a $C=3$ class problem, $x_i$ belongs to the class 0. And let's say the softmax classifier has a prediction of $p_i = [0.3, 0.6, 0.1]$ in the first iteration. When optimizing $L$ with gradient descent, the prediction changes like this:

• [iteration 0] ....... $p_i = [0.3, 0.6, 0.1]$
• [iteration 1] ....... $p_i = [0.4, 0.5, 0.08]$
• [iteration 2] ....... $p_i = [0.55, 0.3, 0.02]$
• ...........................................................
• [iteration n] ....... $p_i = [0.98, 0.001, 0.0001]$

So it seems like gradient descent is trying to increase the value for index 0 and decrease value for the other indices. Based on my understanding, cross-entropy is only evaluated on the correct class, so the other terms will be zeroed out because of the ground truth array $y_i = [1, 0, 0]$:

$L = -\sum_{i=1}^C y_ilog(p_i) = - 1 * log(0.3) = 1.2$

I don't understand the mechanism here. Is it the gradient descent doing these modifications or is it the cross-entropy loss we define or is it the combination of both? How does the prediction score for the correct class increases and the prediction score for the incorrect classes decrease while optimizing?

Answer 1

The value of the loss function depends upon the prediction (which is a function of the input data and the model parameters) and the ground truth. Gradient descent works like this:

• Initialize the model parameters in some manner.
• Using the input data and current model parameters, figure out the loss value of the current network weights and biases.
• Figure out how to update the weights and biases such that the loss value improves.
• Update the weights and biases a certain amount based on the current learning rate.

This process is repeated until convergence (as measured in a few different ways, say improvement against a validation set).

If you want to dig in to how this all works, start with 3Blue1Brown's Neural Network series on YouTube, it's not even that long. It's a good start to get a feel for the concepts before you dig in to the math.

Answer 2

I think that the correct answer is 'both'.

Basically, your goal here is to minimize the error of your prediction. This happens if you predict everything correctly with complete certainty, which is of course generally difficult, but here can be done because we consider only a single data point. Therefore, gradient descent will try try to change your parameters in a direction which increases $p_1$ and decreases $p_2, p_3$ because this is the way you minimize $L$.

But this depends on the loss function you are using. For example if you were using a 'stupid' loss function which is minimized if you misclassify all data points then gradient descent will try to change parameters in such a way as to misclassify as much as it can.