Comparing models using P-value

I have 2 different ML architectures for a translation task, I evaluate them using BLEU score (higher is better)

I've run them 9 times each, yielding the following scores

Architecture 1 | 36.52 | 36.27 | 35.9 | 35.22 | 37.13 | 35.53 | 35.3 | 34.14 | 35

Architecture 2 | 36.85 | 35.64 | 36.37 | 36.82 | 36.74 | 36.46 | 35.77 | 37.31 | 36.68

Means are 35.67 for the first and 36.52 for the second

I want to calculate a P-value that confirms (or not) the superiority of the second model, with alpha = 0.05

• Looks like you have a quite large BLEU variation among scores with the same architecture. Maybe too few instances in your test set? Jul 29 '19 at 14:40
• Yes maybe, I will try on a bigger one if I can Jul 29 '19 at 14:47

I think you can just apply a t-test or Wilcoxon test if you're don't believe in the normality of your observations.

In R:

A <- c(36.52 , 36.27 , 35.9 , 35.22 ,37.13 , 35.53 , 35.3 , 34.14 , 35)
B <- c(36.85 , 35.64 , 36.37 , 36.82 , 36.74,36.46 , 35.77 , 37.31 , 36.68)
wilcox.test(A,B,alternative="less",paired=T)
t.test(A,B,alternative="less",paired=T)

I've done paired tests under the assumption that you did the same folds for each architecture. If that is false, then do unpaired tests.

Edit: You may be interested in a question I posted a few months ago. Cross-validation for model comparison: use the same folds?

• I've done a t-test on sheets using T.TEST(arch2_values,arch1_values,1,1) and got similar results. Could you help me on how to interprete my p-value ? Which is close to 0.03 Jul 29 '19 at 14:57
• @ValentinMacé I don't follow what that line of code does, but $p=0.03$ means that there is only a 3% chance of getting your result or something more extreme if the null hypothesis of equal means is true. Depending on how believable your alternative hypothesis is, that may be low enough or it may not. If you've set your threshold as $\alpha=0.05$ then you've made the decision that anything less than $p=0.05$ is adequate to reject. I think, however, that you did two sided testing. Since you want to prove that one is better rather than they are unequal, consider one-sided testing.
– Dave
Jul 29 '19 at 16:14
• Great, I followed your code example so I guess I did a one-sided test, since I put "less" as alternative parameter, right ? Thanks Jul 30 '19 at 8:15
• It sounds like you did a one-sided test, yes.
– Dave
Jul 30 '19 at 9:50