4
$\begingroup$

I was recently reading the Knowledge Distillation paper, and encountered the term smooth probabilities. The term was used to denote when the logits were divided a temperature. enter image description here

Neural networks typically produce class probabilities by using a softmax output layer that converts the logit, zi, computed for each class into a probability, qi, by comparing zi with the other logits where T is a temperature that is normally set to 1. Using a higher value for T produces a softer probability distribution over classes.

What does that mean intuitively?

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ It means that the distribution is made less 'spiky', or accentuated. It will be readily apparent if you take a random set of input logits and visualize it yourself. This distribution originates from statistical mechanics: en.wikipedia.org/wiki/Boltzmann_distribution $\endgroup$
    – Emre
    Commented Dec 13, 2017 at 17:56

1 Answer 1

Reset to default
7
$\begingroup$

When $T$ gets larger, i.e. $$T \rightarrow \infty $$ the probability distribution resembles the uniform distribution, so $q_i = \frac{1}{n}$. You can find the proof here, as well as some other interesting properties.

Given the previously mentioned fact, I guess he is referring to smoothing in the sense that the noise if left out and what remains are the important patterns of the data, that is the distribution is made less 'spiky' as @Emre mentioned.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.