Multi-class classification configuration

Question

1) What are the appropriate activation and loss functions for multi-class classification problem?

Is it so that:

• Up to 2 classes $\rightarrow$ Binary classification $\rightarrow$ Activation: Sigmoid $\rightarrow$ Loss: binary_crossentropy
• From 3 classes $\rightarrow$ Multi-class classification $\rightarrow$ Activation: Softmax $\rightarrow$ Loss: categorical_crossentropy

If so then...

• 'Dogs and cats' are binary classification but 'dogs, cats and horses' multi-class classification?

• If we have 2 classes then we can't output probability ranges for prediction results?

2) Is it so that for multi-class classification, the last dense layer must have a number of nodes equal to the number of classes?

In the following example there are two classes (dogs and cats) and last dense layer has one node (why not 2 nodes?). If we have 3 or 10 classes then last node should have 3 or 10 nodes?

Answer 1

In my experience the activations are irrelevant to whether the classes are binary or multi-class. Their effectiveness depends on the kind of data you have.

Furthermore, there is one node in the layer because you are doing a binary classification. In other words, when the value of the last node is low or below a threshold the class is 0 and 1 if otherwise(value of last node is high or above a thresold).

And yes, if there are more classes you need as many nodes in the last Dense Layer. You can also you 2 nodes when you only have two classes in your data and predicting the one with the highest score/output.

Answer 2

The reason is simple, when you have a Sigmoid function it will give you number between [0,1] which cn be feed to Cross entropy to get loss. Hence for binary classification with sigmoid one function is enough.

Softmax assumes that each example is a member of exactly one class. Some examples, however, can simultaneously be a member of multiple classes. For such examples:

You may not use Softmax. You must rely on multiple logistic regressions. For example, suppose your examples are images containing exactly one item—a piece of fruit. Softmax can determine the likelihood of that one item being a pear, an orange, an apple, and so on. If your examples are images containing all sorts of things—bowls of different kinds of fruit—then you'll have to use multiple logistic regressions instead.