Why softmax training is more stable

Question

I'm wondering about which activation function will be easier to train with (get better accuracy / smallest loss) - with SoftMax or sigmoid (for multiclass classification problem)

According to: https://www.quora.com/What-are-the-benefits-of-using-a-softmax-function-instead-of-a-sigmoid-function-in-training-deep-neural-networks

Training model for multiclass with SoftMax - the training is more stable vs training with sigmoid

• Why is it true ?
• Is it easier to train model (and get better results) with SoftMax (instead of sigmoid) ?

Answer 1

The short answer is: Yes, it is easier to train a model using SoftMax for multiclass classification compared to sigmoid, and SoftMax generally yields better results (lower loss and higher accuracy).

---

1. SoftMax is Designed for Multiclass Classification

SoftMax produces a normalised probability distribution across all classes, ensuring:

• All outputs are constrained between 0 and 1.
• The probabilities for all classes sum to 1.

This meets the requirements of multiclass classification, where each instance belongs to exactly one class. Sigmoid, by contrast, assigns independent probabilities to each class without ensuring mutual exclusivity, making it unsuitable for multiclass problems.

---

2. Loss Function Compatibility

SoftMax works naturally with the cross-entropy loss function, the standard for classification tasks. Cross-entropy measures the distance between the predicted probability distribution (from SoftMax) and the true distribution (one-hot encoded labels). This compatibility ensures smooth optimisation and convergence.

Sigmoid, when applied independently to each class, does not account for relationships between class probabilities. Using sigmoid for multiclass classification with cross-entropy introduces instability, as the loss function does not properly enforce the mutual exclusivity of classes.

---

3. Training Stability

SoftMax ensures smoother gradients during optimisation:

• The normalisation step reduces the likelihood of vanishing or exploding gradients.
• It allows the optimiser to update parameters in a way that respects the probabilistic structure of the output space.

In contrast, sigmoid’s independent probabilities can lead to conflicting gradient signals during training, especially in multiclass problems, causing slower convergence or unstable training dynamics.

---

4. Empirical Evidence

In practice, models using SoftMax with cross-entropy loss for multiclass classification converge faster and achieve better accuracy compared to models using sigmoid. The normalisation and alignment with the problem’s structure make SoftMax a more effective choice.

---

Why Comparing SoftMax and Sigmoid Can Be Misleading

While SoftMax generalises the logistic sigmoid to multiclass problems, it is not a generalisation of all sigmoid-like functions (e.g., $\tanh$ and $\arctan$). The comparison between SoftMax and sigmoid often arises because both map inputs to constrained ranges, but their purposes differ:

• Sigmoid is primarily suited for:
  • Binary classification.
  • Multi-label classification (independent probabilities for each label).
• SoftMax is specifically designed for multiclass classification (mutually exclusive classes).

Using sigmoid for multiclass classification is a misapplication, leading to unstable training and suboptimal results. Thus, the comparison is somewhat flawed unless the context is clarified.

---

Conclusion

For multiclass classification, SoftMax is the better choice. Its probabilistic outputs, compatibility with cross-entropy loss, and stability during training make it the standard activation function for this task. Sigmoid, while mathematically related, is not designed for multiclass problems and will perform poorly in comparison.