# What is the best way to compare these small distributions?

I have one distribution of size 30.

This are results (ROC-AUC for example) from training a neural network for 30 times in a row with the same hyperparameters but since they are randomly initialized the result is always a little bit different.

Then I train the same network with other hyperparameters and only want to do that for fewer runs. Lets say for 5 runs.

My null hypothesis is that the smaller runs distribution is not smaller than the distribution with 30 runs (one side test).

What kind of statistical significance test would be the best to compare these small distributions?

PS: At the moment I am using Mann Whitney U Test. Is there anything better?

• To clarify, you are comparing two ROC curves? they are not distributions. Sep 11 '20 at 23:11
• No. I have two samples. One with 30 ROC-AUC values (floats) and one with 5. Sep 12 '20 at 12:07

To assess the effects of hyperparameters on the neural network results, you need to eliminate confounding variables. Let's say you have network $$A_t$$ and network $$B_t$$ corresponding to trial/sample $$t$$. The randomly initialized weights of $$A_t$$ should equal $$B_t$$. If you are using a stochastic optimizer (e.g. SGD), you need to ensure that the random instance selection is the same between training network $$A_t$$ and $$B_t$$.