0
$\begingroup$

I have one distribution of size 30.

This are results (ROC-AUC for example) from training a neural network for 30 times in a row with the same hyperparameters but since they are randomly initialized the result is always a little bit different.

Then I train the same network with other hyperparameters and only want to do that for fewer runs. Lets say for 5 runs.

My null hypothesis is that the smaller runs distribution is not smaller than the distribution with 30 runs (one side test).

What kind of statistical significance test would be the best to compare these small distributions?

PS: At the moment I am using Mann Whitney U Test. Is there anything better?

Improve this question
$\endgroup$
2
  • $\begingroup$ To clarify, you are comparing two ROC curves? they are not distributions. $\endgroup$
    – Sean Owen
    Commented Sep 11, 2020 at 23:11
  • $\begingroup$ No. I have two samples. One with 30 ROC-AUC values (floats) and one with 5. $\endgroup$
    – Dieshe
    Commented Sep 12, 2020 at 12:07

1 Answer 1

Reset to default
0
$\begingroup$

To assess the effects of hyperparameters on the neural network results, you need to eliminate confounding variables. Let's say you have network $A_t$ and network $B_t$ corresponding to trial/sample $t$. The randomly initialized weights of $A_t$ should equal $B_t$. If you are using a stochastic optimizer (e.g. SGD), you need to ensure that the random instance selection is the same between training network $A_t$ and $B_t$.

Once you've eliminated confounding variables, then you effectively have paired samples, which you can compare via a Wilcoxon signed-rank test. For small sample sizes, it can be preferable over a paired t-test because 1. you can't verify that the samples are normally distributed and 2. the Wilcoxon test effectively evaluates median rather than mean. The median is more robust against outliers, which are particularly influential if you only have a few samples.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I thought you need matched pairs to do a Wilcoxon signed-rank test. Like human before and after a treatment. In this case I do not have these pairs if I am not wrong. $\endgroup$
    – Dieshe
    Commented Sep 11, 2020 at 19:29
  • $\begingroup$ For Wilcoxon signed-rank test both sample have to have the same size. In my example I have 30 and then only 5. So this does not work. $\endgroup$
    – Dieshe
    Commented Sep 11, 2020 at 19:41
  • 1
    $\begingroup$ Oh I assumed you'd be able to rerun your experiment. If not, I'm not sure there's a way to get meaningful comparisons due to both the confounding variables and the small sample size. I think the best you could do is, as you've recently added in your edit, perform a test like MWW, which is better in this case than t test because the t-test can be more vulnerable to outliers. However, if you are looking to publish the results in some form, you should probably collect data that you can compare more meaningfully. $\endgroup$
    – Ben
    Commented Sep 11, 2020 at 22:13
  • $\begingroup$ Thanks for the Answer Benji Albert. I thought about it and came to the conclusion that a ROC-AUC Metric (for example) must have a specific distribution since it must be between 0 and 1. While other Metrics like MSE can not be smaller than 0 but can be infinite. Isnt that something that could be recognized when doing the significance analysis? $\endgroup$
    – Dieshe
    Commented Sep 12, 2020 at 8:46
  • $\begingroup$ The roc curves indicate how the model performs with varying thresholds of its continuos output. So even though the auc is between 0 and 1, that doesn't provide insight into statistically significant differences between models. You'd still need multiple trials in which you change only a single independent variable to get statistically significant comparisons. $\endgroup$
    – Ben
    Commented Sep 12, 2020 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.