Let us say we have to neural network architectures, A and B and we train $x$ times each of them. Based on the $x$ retrainings, we can calculate $x$ prediction errors for each model, and plot its corresponding distribution. That means, for model A we have an errors density $\mathcal{D}_A$ and for B a density $\mathcal{D}_B$.
Obviously if the mean of the errors of A $\mu_A$ < $\mu_B$ and the standard deviation of the errors of A $\sigma_A < \sigma_B $, I would choose A as my best model. But what if $\mu_A < \mu_B$ but $\sigma_A > \sigma_B$, how do we chose the model.
My question generally is: Given two errors densities $\mathcal{D}_A$ and $\mathcal{D}_B$, what metric compares these two to choose a final model. My simple and maybe incorrect approach I thought about is: make a decision about a reference density $\mathcal{D}$(how you like the errors density to be, for example $\mathcal{N}(0,1)$) and use the KL-divergence to compute the "distance" between each of the A and B densities with the reference one, and choose the model with the smaller distance.
Any ideas ?