Confusion around categorical cross entropy

Question

I understand the binary cross entropy formula for a problem with a single label 0 or 1. If we have more than 2 labels we sum this binary cross entropy over all these classes.

$$ H_{y'}(y) := - \sum_{i} \sum_{c} ({y_{i, c}' \log(y_{i, c}) + (1-y_{i, c}') \log (1-y_{i, c})}) $$

I always thought the above was categorical cross entropy but it turns out its the following:

$$ H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i) $$

I'm pretty sure the two formulas are actually different but I think both would work. I realise the second formula wouldn't work on a problem where the labels are not mutually exclusive (label vector can have multiple 1's), but still I don't see why we can't just use the first formula for all problems whether the labels are mutually exclusive or not? What is the need for the second formula and why is it proffered over the first?

Answer 1

For binary classification, they are the same thing. Imagine $y' \in \{0,1\}$

$$ H_{y'}(y) := - \sum_{i} \sum_{c} ({y_{i, c}' \log(y_{i, c}) + (1-y_{i, c}') \log (1-y_{i, c})}) $$

$$ = - \begin{cases} \sum_{i} \sum_{c} \log(y_{i, c}), &\text{if}& y_{i, c}' = 1 \\ \sum_{i} \sum_{c} \log (1-y_{i, c}), &\text{if}& y_{i, c}' = 0 \end{cases}$$

second formula wouldn't work on a problem where the labels are not mutually exclusive (label vector can have multiple 1's)

This condition is multi-labels which is a different problem. In that case, you shouldn't even use Sigmoid or Softmax. See another answer for multi-labels loss

The second one is used for multi-class classification.

Binary Classification Loss

Cross entropy Loss