Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [knowledge-distillation]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers

Student model overfitting in multiclass classification with knowledge distillation

I'm working on an OCT multiclass classification task using knowledge distillation. My teacher model achieves a solid 97% accuracy and its loss curve demonstrates good stability and generalization. ...
phreak's user avatar
  • 1
4 votes
1 answer

The end-to-end Training Process for Knowledge Distillation

I'm a bit confused on the complete training process for Knowledge Distillation. I was reading the Geoffrey Hinton "Distilling the Knowledge in a Neural Network" 2015 paper and some random ...
Chuu's user avatar
  • 43
0 votes
0 answers

Why are low probabilities problematic for knowledge destilation?

Recently, I have been reading the Knowledge Distillation paper (Distilling the Knowledge in a Neural Network) and I have two main questions: Neural networks typically produce class probabilities by ...
Amir Jalilifard's user avatar
1 vote
1 answer

Distilling a Random Forest to a single DecisionTree, does it make sense?

I stumbled into this blog which shows how a decision tree trained to overfit the predictions of a properly trained random forest model is able to generalize in pretty much the same way as the original ...
amiando's user avatar
  • 35
1 vote
1 answer

Why there is so few research on neural code of artificial neural networks and are there alternatives to the neural code approach?

I feel that the neural code/neural coding (how neurons or biases enode the symbolic concepts of the chains of concepts, e.g. each feature is chain of symbolic functions and their parameters) is the ...
TomR's user avatar
  • 141
1 vote
0 answers

Can I distill knowledge across different neural network frameworks?

I'm interested in using knowledge distillation to compress a large deep learning model to a smaller size so it will run on an embedded device. I've found a number of open source examples for the ...
Sledge's user avatar
  • 254