# KL-divergence to compare ML models

Let us say we have to neural network architectures, A and B and we train $$x$$ times each of them. Based on the $$x$$ retrainings, we can calculate $$x$$ prediction errors for each model, and plot its corresponding distribution. That means, for model A we have an errors density $$\mathcal{D}_A$$ and for B a density $$\mathcal{D}_B$$.

Obviously if the mean of the errors of A $$\mu_A$$ < $$\mu_B$$ and the standard deviation of the errors of A $$\sigma_A < \sigma_B$$, I would choose A as my best model. But what if $$\mu_A < \mu_B$$ but $$\sigma_A > \sigma_B$$, how do we chose the model.

My question generally is: Given two errors densities $$\mathcal{D}_A$$ and $$\mathcal{D}_B$$, what metric compares these two to choose a final model. My simple and maybe incorrect approach I thought about is: make a decision about a reference density $$\mathcal{D}$$(how you like the errors density to be, for example $$\mathcal{N}(0,1)$$) and use the KL-divergence to compute the "distance" between each of the A and B densities with the reference one, and choose the model with the smaller distance.

Any ideas ?

Seems to me a good idea.

KL divergence would give you a raw distance approximation of your distributions, but not all error values might have the same weight of importance: it highly depends on your error calculation method, and some kind of relative error calculation/weightening could be necessary.

In addition to that, Cross Entropy could also be an interesting option to know the "direction" of the distributions' distance.

• Thanks for the answer ... What do you mean by 'depends on your error calculation method'. Assume that I am calculating the MAPE (mean average percentage error), how this will affect my logic ..? Jul 21 at 19:25
• Every method has a different result: MAPE and MSE will not have the same result, neither the same evolution through iterations. As a consequence, the error distribution will not be the same. For instance, the RMSE adds a square to each error that makes high error values more important than MAPE. I recommend using RMSE (good average) or MAE (good use of outliers), rather than MAPE (bad use of high error values & outliers). Jul 22 at 7:50
• If you consider the answers somewhat usefull, don't hesitate to upvote them as acknowledgment :) Jul 22 at 9:06
• I see your point ... thanks Jul 22 at 9:37

For those who are interested in this question, I finally found out some useful metrics that perform well in comparing distributions (and different from KL-divergence): Wasserstein metric, Energy metric, Shannon-Entropy metric, Maximum Mean Discrepancy metric. They are metrics in the sense that they satisfy the properties of a mathematical metric(symmetric for example), while the KL-divergence is not symmetric.

Note that the Shannon-Entropy metric requires density estimation, while the others require only the collected samples.