I'm studying the LLMs course on Coursera. In one video on model distillation, the lecturer says (from the transcript):
Model Distillation is a technique that focuses on having a larger teacher model train a smaller student model. The student model learns to statistically mimic the behavior of the teacher model, either just in the final prediction layer or in the model's hidden layers as well. You'll focus on the first option here. You start with your fine tune LLM as your teacher model and create a smaller LLM for your student model. You freeze the teacher model's weights and use it to generate completions for your training data. At the same time, you generate completions for the training data using your student model. The knowledge distillation between teacher and student model is achieved by minimizing a loss function called the distillation loss. To calculate this loss, distillation uses the probability distribution over tokens that is produced by the teacher model's softmax layer. Now, the teacher model is already fine tuned on the training data. So the probability distribution likely closely matches the ground truth data and won't have much variation in tokens. That's why Distillation applies a little trick adding a temperature parameter to the softmax function. As you learned in lesson one, a higher temperature increases the creativity of the language the model generates. With a temperature parameter greater than one, the probability distribution becomes broader and less strongly peaked. This softer distribution provides you with a set of tokens that are similar to the ground truth tokens.
I've highlighted the sentences I'm having trouble with. The issue is that I understand the teacher model is already fine tuned on the training data, but I don't see why that's an issue:
- You clearly want as good performance for the teacher model as possible. But since the teacher is likely to be very big, the goal is to reduce it so it's more manageable, which is why there's a student model.
- Because you want the teacher to have as good performance as possible, you should apply fine tuning, hence the teacher model is "already fine tuned on the training data". It's therefore not surprising that the teacher model's output closely matches the ground truth data.
- But why is this a problem? You are training a student model to emulate the teacher model (via supervised learning, I presume). The student model has fewer parameters than the teacher, so it should show larger variation with the teacher model's output. If the student model predicts results as good as the teacher model, then there's no need for the teacher in the first place since the student model is strictly superior.
The video goes on to say that the loss function is the difference between the student model's output with temperature > 1 and with temperature = 1. I do not understand why we might want to minimize this difference. It surely does something, but that something does not look like the original goal. Can anyone explain?
