Timeline for What is the best way to compare these small distributions?
Current License: CC BY-SA 4.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 12, 2020 at 14:04 | comment | added | Ben | The roc curves indicate how the model performs with varying thresholds of its continuos output. So even though the auc is between 0 and 1, that doesn't provide insight into statistically significant differences between models. You'd still need multiple trials in which you change only a single independent variable to get statistically significant comparisons. | |
| Sep 12, 2020 at 8:46 | comment | added | Dieshe | Thanks for the Answer Benji Albert. I thought about it and came to the conclusion that a ROC-AUC Metric (for example) must have a specific distribution since it must be between 0 and 1. While other Metrics like MSE can not be smaller than 0 but can be infinite. Isnt that something that could be recognized when doing the significance analysis? | |
| Sep 11, 2020 at 22:13 | comment | added | Ben | Oh I assumed you'd be able to rerun your experiment. If not, I'm not sure there's a way to get meaningful comparisons due to both the confounding variables and the small sample size. I think the best you could do is, as you've recently added in your edit, perform a test like MWW, which is better in this case than t test because the t-test can be more vulnerable to outliers. However, if you are looking to publish the results in some form, you should probably collect data that you can compare more meaningfully. | |
| Sep 11, 2020 at 19:41 | comment | added | Dieshe | For Wilcoxon signed-rank test both sample have to have the same size. In my example I have 30 and then only 5. So this does not work. | |
| Sep 11, 2020 at 19:29 | comment | added | Dieshe | I thought you need matched pairs to do a Wilcoxon signed-rank test. Like human before and after a treatment. In this case I do not have these pairs if I am not wrong. | |
| Sep 11, 2020 at 17:53 | history | answered | Ben | CC BY-SA 4.0 |