2
$\begingroup$

Let us say we have to neural network architectures, A and B and we train $x$ times each of them. Based on the $x$ retrainings, we can calculate $x$ prediction errors for each model, and plot its corresponding distribution. That means, for model A we have an errors density $\mathcal{D}_A$ and for B a density $\mathcal{D}_B$.

Obviously if the mean of the errors of A $\mu_A$ < $\mu_B$ and the standard deviation of the errors of A $\sigma_A < \sigma_B $, I would choose A as my best model. But what if $\mu_A < \mu_B$ but $\sigma_A > \sigma_B$, how do we chose the model.

My question generally is: Given two errors densities $\mathcal{D}_A$ and $\mathcal{D}_B$, what metric compares these two to choose a final model. My simple and maybe incorrect approach I thought about is: make a decision about a reference density $\mathcal{D}$(how you like the errors density to be, for example $\mathcal{N}(0,1)$) and use the KL-divergence to compute the "distance" between each of the A and B densities with the reference one, and choose the model with the smaller distance.

Any ideas ?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Seems to me a good idea.

KL divergence would give you a raw distance approximation of your distributions, but not all error values might have the same weight of importance: it highly depends on your error calculation method, and some kind of relative error calculation/weightening could be necessary.

In addition to that, Cross Entropy could also be an interesting option to know the "direction" of the distributions' distance.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the answer ... What do you mean by 'depends on your error calculation method'. Assume that I am calculating the MAPE (mean average percentage error), how this will affect my logic ..? $\endgroup$
    – outlaw
    Jul 21, 2021 at 19:25
  • 1
    $\begingroup$ Every method has a different result: MAPE and MSE will not have the same result, neither the same evolution through iterations. As a consequence, the error distribution will not be the same. For instance, the RMSE adds a square to each error that makes high error values more important than MAPE. I recommend using RMSE (good average) or MAE (good use of outliers), rather than MAPE (bad use of high error values & outliers). $\endgroup$
    – Nicolas Martin
    Jul 22, 2021 at 7:50
  • $\begingroup$ If you consider the answers somewhat usefull, don't hesitate to upvote them as acknowledgment :) $\endgroup$
    – Nicolas Martin
    Jul 22, 2021 at 9:06
  • $\begingroup$ I see your point ... thanks $\endgroup$
    – outlaw
    Jul 22, 2021 at 9:37
1
$\begingroup$

For those who are interested in this question, I finally found out some useful metrics that perform well in comparing distributions (and different from KL-divergence): Wasserstein metric, Energy metric, Shannon-Entropy metric, Maximum Mean Discrepancy metric. They are metrics in the sense that they satisfy the properties of a mathematical metric(symmetric for example), while the KL-divergence is not symmetric.

Note that the Shannon-Entropy metric requires density estimation, while the others require only the collected samples.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.