1
$\begingroup$

I am talking about a scenario where you first train a "huge" Neural Network and then try to scale it down without sacrificing much of the accuracy. I am not talking about quantization of weights, biases, etc. My thought comes from the perspective of coarse-grained molecular dynamics in physics. Which is like smoothing out the energy landscape. Here in the neural network context too we have an "energy landscape"(loss landscape?). So is there any works I can look up which has done something similar?

$\endgroup$

1 Answer 1

1
$\begingroup$

What you are looking for is called Knowledge Distillation which means learning the behaviour of a large model with a smaller one, usually via a training paradigm called Teacher-Student.

It is pretty straightforward. Imagine you have a huge model that you trained by any means. This model can predict inputs for you in a way that it is supposed to do. Now you want to reduce this huge model to a smaller one.

You can feed your data to that model and get all the outputs. Then this pairs of $(x, y)$s can be used as "labels data" for training a smaller model (right?).

Explanation is easy as I find it pretty clear (specially if you have background in ML). If it is not, then comment so I update my answer with real demonstration in Python.

$\endgroup$
1
  • $\begingroup$ thank you for the answer. I will first read about knowledge distillation. $\endgroup$
    – dexterdev
    Mar 12, 2022 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.