# what does smooth/soft probablity mean?

I was recently reading the Knowledge Distillation paper, and encountered the term smooth probabilities. The term was used to denote when the logits were divided a temperature.

Neural networks typically produce class probabilities by using a softmax output layer that converts the logit, zi, computed for each class into a probability, qi, by comparing zi with the other logits where T is a temperature that is normally set to 1. Using a higher value for T produces a softer probability distribution over classes.

What does that mean intuitively?

• It means that the distribution is made less 'spiky', or accentuated. It will be readily apparent if you take a random set of input logits and visualize it yourself. This distribution originates from statistical mechanics: en.wikipedia.org/wiki/Boltzmann_distribution – Emre Dec 13 '17 at 17:56

When $T$ gets larger, i.e. $$T \rightarrow \infty$$ the probability distribution resembles the uniform distribution, so $q_i = \frac{1}{n}$. You can find the proof here, as well as some other interesting properties.