1
$\begingroup$

I have 2 different ML architectures for a translation task, I evaluate them using BLEU score (higher is better)

I've run them 9 times each, yielding the following scores

Architecture 1 | 36.52 | 36.27 | 35.9 | 35.22 | 37.13 | 35.53 | 35.3 | 34.14 | 35

Architecture 2 | 36.85 | 35.64 | 36.37 | 36.82 | 36.74 | 36.46 | 35.77 | 37.31 | 36.68

Means are 35.67 for the first and 36.52 for the second

I want to calculate a P-value that confirms (or not) the superiority of the second model, with alpha = 0.05

Thanks in advance

Improve this question
$\endgroup$
2
  • $\begingroup$ Looks like you have a quite large BLEU variation among scores with the same architecture. Maybe too few instances in your test set? $\endgroup$
    – Erwan
    Commented Jul 29, 2019 at 14:40
  • $\begingroup$ Yes maybe, I will try on a bigger one if I can $\endgroup$
    – Valentin Macé
    Commented Jul 29, 2019 at 14:47

1 Answer 1

Reset to default
2
$\begingroup$

I think you can just apply a t-test or Wilcoxon test if you're don't believe in the normality of your observations.

In R:

A <- c(36.52 , 36.27 , 35.9 , 35.22 ,37.13 , 35.53 , 35.3 , 34.14 , 35)
B <- c(36.85 , 35.64 , 36.37 , 36.82 , 36.74,36.46 , 35.77 , 37.31 , 36.68)
wilcox.test(A,B,alternative="less",paired=T)
t.test(A,B,alternative="less",paired=T)

I've done paired tests under the assumption that you did the same folds for each architecture. If that is false, then do unpaired tests.

Edit: You may be interested in a question I posted a few months ago. Cross-validation for model comparison: use the same folds?

Improve this answer
$\endgroup$
4
  • $\begingroup$ I've done a t-test on sheets using T.TEST(arch2_values,arch1_values,1,1) and got similar results. Could you help me on how to interprete my p-value ? Which is close to 0.03 $\endgroup$
    – Valentin Macé
    Commented Jul 29, 2019 at 14:57
  • $\begingroup$ @ValentinMacé I don't follow what that line of code does, but $p=0.03$ means that there is only a 3% chance of getting your result or something more extreme if the null hypothesis of equal means is true. Depending on how believable your alternative hypothesis is, that may be low enough or it may not. If you've set your threshold as $\alpha=0.05$ then you've made the decision that anything less than $p=0.05$ is adequate to reject. I think, however, that you did two sided testing. Since you want to prove that one is better rather than they are unequal, consider one-sided testing. $\endgroup$
    – Dave
    Commented Jul 29, 2019 at 16:14
  • $\begingroup$ Great, I followed your code example so I guess I did a one-sided test, since I put "less" as alternative parameter, right ? Thanks $\endgroup$
    – Valentin Macé
    Commented Jul 30, 2019 at 8:15
  • $\begingroup$ It sounds like you did a one-sided test, yes. $\endgroup$
    – Dave
    Commented Jul 30, 2019 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.