Recently, I have been reading the Knowledge Distillation paper (Distilling the Knowledge in a Neural Network) and I have two main questions:
Neural networks typically produce class probabilities by using a softmax output layer that converts the logit, zi, computed for each class into a probability, qi, by comparing zi with the other logits where T is a temperature that is normally set to 1. Using a higher value for T produces a softer probability distribution over classes.
- I know that softer distribution means being less skippy as it was already responded to in the following post what does smooth/soft probablity mean?. But what is the problem of skippy distribution?
- These low probabilities are produced by the Teacher Model and they reflect the confidence of the Teacher Model in its predictions (i.e.; 98% class A and 2% class B, is a better model than the one which gives 75% class A and 25% class B since the first model is more confident in its correct prediction) and how it "thinks", whether we like them or not. Using the temperature T changes the output of the Teacher Model which should be a problem because we are actually changing the output of a model we previously tried so hard to train. Why is not changing model output a problem?