So I have the following problem: I realized (while writing my master thesis) that I am still not sure/have vague descriptions of some of the machine learning principles. I already asked one question regard definitions that can be found here.

Now I stumbled over another definition Problem. Here is an excerpt from my thesis (this is in particular about neural-network classification):

If the classes are mutually exclusive (i.e. if a sample $x^{j} = C_{0}$, $x^{j} \neq C_{i}\setminus~C_{0}$ ), the probabilities of all classes add up to one like \begin{equation} \sum_{i} P(x^{j}=C_{i}) = 1. \end{equation} In this case the best practice is to use a softmax activation function for the output neurons. If the classes are not mutually exclusive it would suffice to use a sigmoid output activation function, as the sigmoid function gets independent probabilities for each class \begin{equation} \sum_{i} P(x^{j}=C_{i}) \geq 1. \end{equation}

I already found the following link regarding this topic. However I know that in practise if you don't use softmax activation function in your output layer, the value can be larger than 1 but can a probability be larger 1? Isn't that against its definition?

Is a non-mutual classification really a common case? Can somebody may be linking some cases (paper?) were they needed non-mutual classification?

  • $\begingroup$ You may be interested in this paper -- sparse softmax. $\endgroup$ – Logan Sep 30 '19 at 2:05

You are correct: a probability cannot be larger than 1.

At the final layer, the activations (also known as logits), are passed through the final a softmax function in order to fulfill this constraint. The standard neural network does not have an implicit mechanism by it can ensure that constraint is met during training etc.

By non-mutual classification, we could be talking about something like classifying cat and dog images - in which case the label for each image either cat or dog. So they are mutually exclusive. This is a very common case - almost any form of image classification falls into this category.

You do not use a sigmomid function (or any other non-linearity for that matter) after the final layer, as there are no neurons following them, making a non-linearity somewhat redundant. Using a non-linearity for the purpose of fitting a non-linear model is different to the purpose of a final softmax function. This has exactly the purpose of scaling the final logits/activations into the nice range of [0, 1] that can be interpreted as probabilities. That allows us to make simple rules on how to classify the outputs - e.g. if p = [0.51, 0.49] then that sample was a cat, whereas p =[0.49, 0.51] is a dog.

I used those values in the example to highlight a further point; namely that you cannot interpret them as pure probabilities. Those examples don't mean the model was really unsure in both cases because all four "probabilities" we close to 0.5. The model gives more weight to the option is believes is correct - the relative magnitudes of those values are not directly interpretable.

  • $\begingroup$ You do not use a sigmomid function (or any other non-linearity for that matter) after the final layer, as there are no neurons following them, making a non-linearity somewhat redundant. Can you cite some sources? For me intuitionally no-hidden layer sigmoid can approximate many non-linear functions $\endgroup$ – DuttaA Aug 23 '18 at 15:23
  • $\begingroup$ I have looked for citations on this before and, unfortunately, failed. It seems to be a heuristic for now. The same goes for whether to apply dropout before or after the activation. We update the weights (neurons) via backprop, and so putting another non-linearity between the model's final output and where the errors are computed seems to me like adding a pointless obstacle that the model will then have account for. I would always just add another FC-layer after the non-linearities so the neurons provide "the final say", so to speak. $\endgroup$ – n1k31t4 Aug 23 '18 at 15:32
  • $\begingroup$ If i remember correctly 2 nodes can easily approximate a sine function, maybe not useful but it still adds some non-linearity, definitely not useful in case of multi-class classification unless the data is in a series of square pulses $\endgroup$ – DuttaA Aug 23 '18 at 15:34
  • $\begingroup$ Thank you for your answer, however I think you misunderstood my question. Maybe I have to be clearer: It was not about using a sigmoid activation function after the last layer but instead of the softmax in the last layer. I know about the mutual exclusive case, however I was interested in the case were the classes are non-mutually exclusive, how the math would be in that case and if anyone knows of such a case existing in practise. As you said there is no implicit mechanic in neural networks to model exactly probabilities. But couldn't you, e.g. relax the arg max in the output $\endgroup$ – dodo Aug 24 '18 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.