# Non-mutually exclusive classification sum of probabilities

So I have the following problem: I realized (while writing my master thesis) that I am still not sure/have vague descriptions of some of the machine learning principles. I already asked one question regard definitions that can be found here.

Now I stumbled over another definition Problem. Here is an excerpt from my thesis (this is in particular about neural-network classification):

If the classes are mutually exclusive (i.e. if a sample $x^{j} = C_{0}$, $x^{j} \neq C_{i}\setminus~C_{0}$ ), the probabilities of all classes add up to one like $$\sum_{i} P(x^{j}=C_{i}) = 1.$$ In this case the best practice is to use a softmax activation function for the output neurons. If the classes are not mutually exclusive it would suffice to use a sigmoid output activation function, as the sigmoid function gets independent probabilities for each class $$\sum_{i} P(x^{j}=C_{i}) \geq 1.$$

I already found the following link regarding this topic. However I know that in practise if you don't use softmax activation function in your output layer, the value can be larger than 1 but can a probability be larger 1? Isn't that against its definition?

Is a non-mutual classification really a common case? Can somebody may be linking some cases (paper?) were they needed non-mutual classification?

• You may be interested in this paper -- sparse softmax. Sep 30, 2019 at 2:05
• The output of sigmoid and softmax can't be more than 1. The sum of several sigmoid outputs could be; it's not a probability. But it also need not be >= 1. The sum of N probabilities must be in [0,N] though. Apr 27, 2020 at 17:48

You are correct: a probability cannot be larger than 1.

At the final layer, the activations (also known as logits), are passed through the final a softmax function in order to fulfill this constraint. The standard neural network does not have an implicit mechanism by it can ensure that constraint is met during training etc.

By non-mutual classification, we could be talking about something like classifying cat and dog images - in which case the label for each image either cat or dog. So they are mutually exclusive. This is a very common case - almost any form of image classification falls into this category.

You do not use a sigmomid function (or any other non-linearity for that matter) after the final layer, as there are no neurons following them, making a non-linearity somewhat redundant. Using a non-linearity for the purpose of fitting a non-linear model is different to the purpose of a final softmax function. This has exactly the purpose of scaling the final logits/activations into the nice range of [0, 1] that can be interpreted as probabilities. That allows us to make simple rules on how to classify the outputs - e.g. if p = [0.51, 0.49] then that sample was a cat, whereas p =[0.49, 0.51] is a dog.

I used those values in the example to highlight a further point; namely that you cannot interpret them as pure probabilities. Those examples don't mean the model was really unsure in both cases because all four "probabilities" we close to 0.5. The model gives more weight to the option is believes is correct - the relative magnitudes of those values are not directly interpretable.

• You do not use a sigmomid function (or any other non-linearity for that matter) after the final layer, as there are no neurons following them, making a non-linearity somewhat redundant. Can you cite some sources? For me intuitionally no-hidden layer sigmoid can approximate many non-linear functions Aug 23, 2018 at 15:23
• I have looked for citations on this before and, unfortunately, failed. It seems to be a heuristic for now. The same goes for whether to apply dropout before or after the activation. We update the weights (neurons) via backprop, and so putting another non-linearity between the model's final output and where the errors are computed seems to me like adding a pointless obstacle that the model will then have account for. I would always just add another FC-layer after the non-linearities so the neurons provide "the final say", so to speak. Aug 23, 2018 at 15:32
• If i remember correctly 2 nodes can easily approximate a sine function, maybe not useful but it still adds some non-linearity, definitely not useful in case of multi-class classification unless the data is in a series of square pulses Aug 23, 2018 at 15:34
• Thank you for your answer, however I think you misunderstood my question. Maybe I have to be clearer: It was not about using a sigmoid activation function after the last layer but instead of the softmax in the last layer. I know about the mutual exclusive case, however I was interested in the case were the classes are non-mutually exclusive, how the math would be in that case and if anyone knows of such a case existing in practise. As you said there is no implicit mechanic in neural networks to model exactly probabilities. But couldn't you, e.g. relax the arg max in the output
– dodo
Aug 24, 2018 at 7:55

Just from the probabilistic side of things: When the classes are not mutually exclusive, the events $$x^j=C_i$$ are not disjoint, so in general (if every example gets some label),

$$1 = P(x^j=C_0 \vee \dotsb \vee x^j=C_t) \lneq \sum_i P(x^j=C_i).$$

the value can be larger than 1 but can a probability be larger 1?

is "the value (of the sum) is not a probability, it's a sum of probabilities."

Is a non-mutual classification really a common case? Can somebody may be linking some cases (paper?) were they needed non-mutual classification?

This is commonly known as "multi-label classification." Examples include topics/genres/themes, where a given item may include more than one. I'll defer to the question you linked to provide other links.

Note that the probability being at least 1 above relied on every item receiving at least one label. In cases where that's not the case, the sum may be less than 1. And the outputs of a neural network (when individually sigmoid-activated) may sum to less than 1 even when every item receives a label, from imperfect calibration.

Finally, I'd just like to say that the notation $$x^j=C_0,\ x^j\neq C_i\setminus C_0$$ seems off to me; using $$\in$$ and $$\notin$$ seem better.

the value can be larger than 1 but can a probability be larger 1? Isn't that against its definition?

Speaking in a very simple language how a model(NN) works - It doesn't know if it is a probability Or a number. It only knows that it has to minimise the Loss to match the output.

I see no reason why can't an output become > 1 if we don't use Sigmoid/Softmax and provide the model Y =[1, 0, 0, 1, 1] (For 5 data points).
It can end ~1/-ve e.g. 1.05, -0.05 etc.
We use Sigmoid Or Softmax to covert this value to Class. Softmax has the additional property to magnify bigger values and stretch the difference in the input.

Is a non-mutual classification really a common case?

It's the case of multi-label Classification.
A 10-Class image with [Cat, Dog, Flower] in one image will have Y= [1, 0, 0, 1, 1, 0, 0, 0, 0, 0]

I hope, I have not over-simplified things.