I'm studying the LLMs course on Coursera. In one video on model distillation, the lecturer says (from the transcript):

Model Distillation is a technique that focuses on having a larger teacher model train a smaller student model. The student model learns to statistically mimic the behavior of the teacher model, either just in the final prediction layer or in the model's hidden layers as well. You'll focus on the first option here. You start with your fine tune LLM as your teacher model and create a smaller LLM for your student model. You freeze the teacher model's weights and use it to generate completions for your training data. At the same time, you generate completions for the training data using your student model. The knowledge distillation between teacher and student model is achieved by minimizing a loss function called the distillation loss. To calculate this loss, distillation uses the probability distribution over tokens that is produced by the teacher model's softmax layer. Now, the teacher model is already fine tuned on the training data. So the probability distribution likely closely matches the ground truth data and won't have much variation in tokens. That's why Distillation applies a little trick adding a temperature parameter to the softmax function. As you learned in lesson one, a higher temperature increases the creativity of the language the model generates. With a temperature parameter greater than one, the probability distribution becomes broader and less strongly peaked. This softer distribution provides you with a set of tokens that are similar to the ground truth tokens.

I've highlighted the sentences I'm having trouble with. The issue is that I understand the teacher model is already fine tuned on the training data, but I don't see why that's an issue:

  • You clearly want as good performance for the teacher model as possible. But since the teacher is likely to be very big, the goal is to reduce it so it's more manageable, which is why there's a student model.
  • Because you want the teacher to have as good performance as possible, you should apply fine tuning, hence the teacher model is "already fine tuned on the training data". It's therefore not surprising that the teacher model's output closely matches the ground truth data.
  • But why is this a problem? You are training a student model to emulate the teacher model (via supervised learning, I presume). The student model has fewer parameters than the teacher, so it should show larger variation with the teacher model's output. If the student model predicts results as good as the teacher model, then there's no need for the teacher in the first place since the student model is strictly superior.

The video goes on to say that the loss function is the difference between the student model's output with temperature > 1 and with temperature = 1. I do not understand why we might want to minimize this difference. It surely does something, but that something does not look like the original goal. Can anyone explain?


1 Answer 1


With a smaller amount of information for training the student you want to utilize as much information as you can. The data points with very small probabilities are too close to zero to be utilized. You will end up training on a smaller amount of data with a higher confidence of class. Increasing the temperature helps you include more diverse samples in the student-training.

See this explanation for an example of values


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.