A cross-entropy loss explanation in simple words

Suppose I build a FNN model. The last layer is a classification layer with softmax activation.
A cross-entropy loss is used to classify a problems, such as logistic regression. How would I calculate the cross entropy loss ?

But I can not fully understand the cross-entropy loss Please any convenient explanation

The cross entropy formula takes in two distributions, $$p(x)$$, the true distribution, and $$q(x)$$, the estimated distribution, defined over the discrete variable $$x$$ and is given by

$$H(p,q) = -\sum_{\forall x} p(x) \log(q(x))$$

For a neural network, the calculation is independent of the following:

• What kind of layer was used.

• What kind of activation was used - although many activations will not be compatible with the calculation because their outputs are not interpretable as probabilities (i.e., their outputs are negative, greater than 1, or do not sum to 1). Softmax is often used for multiclass classification because it guarantees a well-behaved probability distribution function.

For a neural network, you will usually see the equation written in a form where $$\mathbf{y}$$ is the ground truth vector and $$\mathbf{\hat{y}}$$ (or some other value taken direct from the last layer output) is the estimate. For a single example, it would look like this:

$$L = - \mathbf{y} \cdot \log(\mathbf{\hat{y}})$$

where $$\cdot$$ is the vector dot product.

Your example ground truth $$\mathbf{y}$$ gives all probability to the first value, and the other values are zero, so we can ignore them, and just use the matching term from your estimates $$\mathbf{\hat{y}}$$

$$L = -(1\times log(0.1) + 0 \times \log(0.5) + ...)$$

$$L = - log(0.1) \approx 2.303$$

That means, the loss would be same no matter if the predictions are $$[0.1, 0.5, 0.1, 0.1, 0.2]$$ or $$[0.1, 0.6, 0.1, 0.1, 0.1]$$?
You will often see this equation averaged over all examples as a cost function. It is not always strictly adhered to in descriptions, but usually a loss function is lower level and describes how a single instance or component determines an error value, whilst a cost function is higher level and describes how a complete system is evaluated for optimisation. A cost function based on multiclass log loss for data set of size $$N$$ might look like this: