Distillation seems to be a general technique to reduce the size of NLP/NN models. Can anyone help me to understand intuition and how does it work?
Someone, such as Albert Einstein, spends his/her entire life coming up with a solution to a problem, such as Special Relativity. Then within a single class, maybe an hour and a half, a teacher teaches you and you now have the same knowledge. That's basically what happens in distilling knowledge. The teacher model does the hard work by learning the relationship between inputs and outputs. The student model learns that relationship from the teacher network. A prime example is DeiT in which the authors trained vision transformer with CNN teacher and achieved competitive performance with the original vision transformer ViT that needed about 300M images for pretraining.
Distillation (sometimes also called knowledge distillation or teacher-student training) is the technique of 'distilling' knowledge from a large neural network into a smaller neural network, generally allowing you achieve a similar performance (i.e. accuracy wise) with a smaller network. An example of such a model would be the DistilBERT model, which uses distilled knowledge from the BERT model.
The way this technique works is that you have two models, one being the large model (also called teacher model) and a smaller model (also called the student model). During the training phase the student model tries to predict values and changes its weights based on a loss which combines both the error of the prediction of the student model and the error between the prediction of the teacher model and student model. The idea is that in this way the teacher model helps guide the student model to make the correct predictions.
Some additional links with additional info on knowledge distillation are this documentation page from the
Distiller package and this practical example from Weights & Biases.