1
$\begingroup$

I have 2 different ML architectures for a translation task, I evaluate them using BLEU score (higher is better)

I've run them 9 times each, yielding the following scores

Architecture 1 | 36.52 | 36.27 | 35.9 | 35.22 | 37.13 | 35.53 | 35.3 | 34.14 | 35

Architecture 2 | 36.85 | 35.64 | 36.37 | 36.82 | 36.74 | 36.46 | 35.77 | 37.31 | 36.68

Means are 35.67 for the first and 36.52 for the second

I want to calculate a P-value that confirms (or not) the superiority of the second model, with alpha = 0.05

Thanks in advance

$\endgroup$
2
  • $\begingroup$ Looks like you have a quite large BLEU variation among scores with the same architecture. Maybe too few instances in your test set? $\endgroup$
    – Erwan
    Jul 29, 2019 at 14:40
  • $\begingroup$ Yes maybe, I will try on a bigger one if I can $\endgroup$ Jul 29, 2019 at 14:47

1 Answer 1

2
$\begingroup$

I think you can just apply a t-test or Wilcoxon test if you're don't believe in the normality of your observations.

In R:

A <- c(36.52 , 36.27 , 35.9 , 35.22 ,37.13 , 35.53 , 35.3 , 34.14 , 35)
B <- c(36.85 , 35.64 , 36.37 , 36.82 , 36.74,36.46 , 35.77 , 37.31 , 36.68)
wilcox.test(A,B,alternative="less",paired=T)
t.test(A,B,alternative="less",paired=T)

I've done paired tests under the assumption that you did the same folds for each architecture. If that is false, then do unpaired tests.

Edit: You may be interested in a question I posted a few months ago. Cross-validation for model comparison: use the same folds?

$\endgroup$
4
  • $\begingroup$ I've done a t-test on sheets using T.TEST(arch2_values,arch1_values,1,1) and got similar results. Could you help me on how to interprete my p-value ? Which is close to 0.03 $\endgroup$ Jul 29, 2019 at 14:57
  • $\begingroup$ @ValentinMacé I don't follow what that line of code does, but $p=0.03$ means that there is only a 3% chance of getting your result or something more extreme if the null hypothesis of equal means is true. Depending on how believable your alternative hypothesis is, that may be low enough or it may not. If you've set your threshold as $\alpha=0.05$ then you've made the decision that anything less than $p=0.05$ is adequate to reject. I think, however, that you did two sided testing. Since you want to prove that one is better rather than they are unequal, consider one-sided testing. $\endgroup$
    – Dave
    Jul 29, 2019 at 16:14
  • $\begingroup$ Great, I followed your code example so I guess I did a one-sided test, since I put "less" as alternative parameter, right ? Thanks $\endgroup$ Jul 30, 2019 at 8:15
  • $\begingroup$ It sounds like you did a one-sided test, yes. $\endgroup$
    – Dave
    Jul 30, 2019 at 9:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.