In Distilling a Neural Network Into a Soft Decision Tree the authors mention
exponentially decaying running average of the actual probabilities with a time window
but provide no formula for it.
What does it mean speaking formal?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Interesting paper. Note that in this question you are referring to how the tree is built in order to help distile the knowledge in a NN $\endgroup$– Carlos MouganCommented Dec 19, 2020 at 9:50
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This is likely the Exponential Moving Average function. In simplest form it is
ma_new = alpha * new_sample + (1-alpha) * ma_old
Where alpha is the parameter that controls rate of decay, between 0.0-1.0.
