I'm creating a deep network with 3 hidden layers for classification of the MNIST dataset. The network architecture is something like:

784 input nodes -> 150 hidden nodes (layer 1) -> 125 hidden nodes (layer 2) -> 100 hidden nodes (layer 3) -> 10 output nodes (1 for each possible digit)

For the output layer, I need to decide on an activation function. I've previously used the sigmoid activation function, but it seems many people are using softmax in this particular implementation. I'd like to know which is best suited for the task and why.

Many thanks

  • 1
    $\begingroup$ Just try both. You don't have anything to lose if you do it while you're asleep. Should not take too long to train these models. The following link recommends to use softmax for multi-class labeling dataaspirant.com/2017/03/07/…. $\endgroup$ – JahKnows Mar 21 '18 at 10:29
  • 1
    $\begingroup$ stats.stackexchange.com/questions/233658/…, here is another link from SE which discusses this question. $\endgroup$ – JahKnows Mar 21 '18 at 10:30

This choice mainly depends on what your output represents. Given a vector $\mathbf{x}$, the sigmoid function is given by $$ \sigma(x_i) = \frac{\exp(x_i)}{1 + \exp(x_i)} $$ while the softmax is given by $$ \mathrm{softmax}(x_i) = \frac{\exp(x_i)}{\sum_{j=0}^N \exp(x_j)}$$ A key difference is that the output of the sigmoid function applied to $x_i$ only depends on $x_i$. The output of the softmax function depends on all elements of the vector $\mathbf{x}$.

The sigmoid will squash each $x_i$ into the range $(0, 1)$, which enables you to interpret $\sigma(x_i)$ as the probability of $x_i$. But, if you have multiple classes, e.g. 0-9 in MNIST, each probability is independent, that means you could have probability $p=0.9$ for the digit 1 and $p=0.5$ for the digit 3. This is undesirable if you want to distinguish between multiple classes. However, if multiple classes can appear at the same time, then sigmoid is well suited.

Now, the softmax is basically a sigmoid function which is normalized such that $\sum_{j=0}^N \mathrm{softmax}(x_j) = 1$. This means that the output of a softmax layer is a valid probability mass function, i.e. the output of your neural network is the probability of the input belonging to a certain class.

So, if you want to classify between e.g. the 10 digits as in MNIST, then softmax is the way to go. If the classes are independent, i.e. it is possible that an image shows a cat and a dog at the same time, then sigmoid is the way to go.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.