0
$\begingroup$

Recently, I have been reading the Knowledge Distillation paper (Distilling the Knowledge in a Neural Network) and I have two main questions:

Neural networks typically produce class probabilities by using a softmax output layer that converts the logit, zi, computed for each class into a probability, qi, by comparing zi with the other logits where T is a temperature that is normally set to 1. Using a higher value for T produces a softer probability distribution over classes.

  1. I know that softer distribution means being less skippy as it was already responded to in the following post what does smooth/soft probablity mean?. But what is the problem of skippy distribution?
  2. These low probabilities are produced by the Teacher Model and they reflect the confidence of the Teacher Model in its predictions (i.e.; 98% class A and 2% class B, is a better model than the one which gives 75% class A and 25% class B since the first model is more confident in its correct prediction) and how it "thinks", whether we like them or not. Using the temperature T changes the output of the Teacher Model which should be a problem because we are actually changing the output of a model we previously tried so hard to train. Why is not changing model output a problem?
Improve this question
$\endgroup$
1
  • $\begingroup$ the decision boundary is the same, but the confidence is different... if you initialize the last layer of a classification network with high weights, then the loss will be high as the network is not well calibrated (the problem is that you want the net to do errors in uncertain situations, not when it has high confidence) $\endgroup$
    – Alberto Sinigaglia
    Nov 6 at 13:53

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.