significance test and sample size estimation for classifiers

Question

What is the test to tell if e.g. an F1 score of 0.69 for classifier A and 0.72 for classifier B is truly different and not just by chance? (for mean-values one would use a "t-test" and obtain a "p-value"). I have access to the underlying data and not only to the F1 scores.

... and how can one estimate the sample size needed in the test-set in order not to miss a true difference between the F1 scores? (as in the example above) (for mean-values one would use a "power analysis"). Or in other words, if I want to know which classifier (A or B) is truly better (to a certain significance level): how many test cases do I need?

Google just returns some research papers but I would need some type of established standard methods for the sample size and significance test (ideally implemented as a python package).

--- EDIT ---

Thanks for pointing out this post in the comments - it points in the right direction but unfortunately does not solve my two related problems as stated above.

Answer 1

I think there are two primary possibilities for answers, so I'll try to address both of them:

(Recommended) If you have multiple independent trials per classifier:

This is a lot simpler than if you have only one trial per classifer (and much better). If you have multiple $F_1$ scores per classifier, then you can simply conduct a paired t-test or a Wilcoxon signed rank test. You'd probably go with Wilcoxon if you don't have many trials to approximate a normal distribution via the central limit theorem or if for whatever reason your $F_1$ scores are not normally distributed.

If you only have one trial per classifier:

In this situation, we are going to find a way to express $F_1$ in terms of $F_1\pm \delta_{F_1}$ to see if the two scores overlap given some confidence level.

Step 1: Decomposition

We can first decompose the metric we are trying to calculate: $$F_1\equiv2\frac{precision\cdot recall}{precision+recall}\equiv2\frac{PPV\cdot TPR}{PPV+TPR}$$

We are looking to find the uncertainty in the $F_1$ score, so we first need to find the uncertainties in the values composing it. To do this, we can consider the precision and recall metrics in terms of Bernoulli trials by partitioning the instances into $P$ and $N$, positives and negatives. Let's also assume $P+N\gg1$.

---

Edit to answer the question about determining sample sizes:

First, choose a maximum width $w$ for your confidence interval in terms of percentage points (i.e. choose a margin of error MoE and double it to account for both sides of the mean). Then, using our notion of Bernoulli trials, we can determine your minimum sample size based on the z-score for your confidence interval as well as the width (MoE):

$$n\ge \frac{4z^2(p(1-p))}{w^2} \because z\sqrt{\frac{p(1-p)}{n}}\le MoE$$

The safest estimate would be to choose $p=0.5$ which would maximize $n$ for a given $z$ and $MoE$. Realistically though, this would be an overestimate for the required $n$ because you probably don't have a random classifier.

As an example, if you wanted an MoE of 10% for a CI of 95%, you'd need: $n\ge 97$ assuming worst case $p=0.5$.

---

When we consider Bernoulli trials, we can talk in terms of success/failure. So the number of successful $P$, which would be $TP$, can be considered a percentage of the total number of $P$ instances, which is simply the true positive rate $TPR$. The same goes for $TN$ and $N$, being $TNR$. In other words:

$$TPR=\frac{TP}{P}=\frac{TP}{TP+FN}\enspace \enspace, \enspace\enspace TNR=\frac{TN}{N}=\frac{TN}{TN+FP}$$

Step 2: Binomial Proportion Confidence Intervals

Now, we can develop binomial proportion confidence intervals per $P$ and $N$. There are many formulas to do this, but I am going to stick with probably the most basic one. That way, we make sure that our result assumes the form $TPR\pm \delta_{TPR}$.

Now let's pick a confidence level:

• If you want a $99\%$, then $z_{0.01/2}\approx2.576$
• If you want $95\%$, then $z_{0.05/2}\approx 1.96$ confidence level

We can calculate the individual confidence intervals:

$$TPR\pm z\sqrt{\frac{TPR(1-TPR)}{P}}\enspace \enspace, \enspace\enspace TNR\pm z\sqrt{\frac{TNR(1-TNR)}{N}}$$

To get $PPV$, we can first define $TP=P(TPR)$ and $FP=N(1-TNR)$. Then: $$ PPV=\frac{TP}{TP+FP}=\frac{P(TPR)}{P(TPR)+N(1-TNR)} $$

Step 3: Propagate PPV Uncertainties

We want to find $PPV\pm \delta_{PPV}$, so we need to propagate the uncertainties from $TPR$ and $TNR$. Because $P$ and $N$ are determined by the dataset rather than your model (and they are exact "measurements"), they are without uncertainties:

$$ PPV\pm \frac{P(TPR\pm \delta_{TPR})}{P(TPR\pm \delta_{TPR})+N-N(TNR\pm \delta_{TNR})}=\frac{X}{Y} $$ Let's first find the individual components $X$ and $Y$, keeping in mind that when uncertain values add, their uncertainties add as well: $$ X=P(TPR)\pm P(\delta_{TPR}) $$ $$ Y=(N+P(TPR)+N(TNR))\pm (P(\delta_{TPR})+N(\delta_{TNR})) $$ $$ = \frac{P(TPR)\pm P(\delta_{TPR})}{P(TPR\pm \delta_{TPR})+N-N(TNR\pm \delta_{TNR})}$$

Now we can divide $X$ by $Y$. To do so, we need to keep in mind that the resulting uncertainty is the sum of their fractional uncertainties, like so: $$ PPV=\frac{X}{Y}\enspace \enspace , \enspace \enspace \frac{\delta_{PPV}}{|PPV|}=\frac{\delta_X}{|X|}+\frac{\delta_Y}{|Y|}\Rightarrow \delta{PPV}=\frac{X}{Y}\left(\frac{\delta_X}{|X|}+\frac{\delta_Y}{|Y|}\right) $$

For simplicity, I'll write $PPV$ because we know that $$PPV\pm \delta_{PPV}\equiv \frac{X}{Y}\pm\frac{X}{Y}\left(\frac{\delta_X}{|X|}+\frac{\delta_Y}{|Y|}\right)$$

(Finally) Step 4: Propagate TPR and PPV Uncertainties

Just as we did with $PPV$, we need to propagate the uncertainties for the $F_1$ score calculation. Fortunately, it's the same process as the uncertainties for multiplication combine in the same manner as division:

$$ F_1 = 2\frac{PPV\cdot TPR}{PPV+TPR} = 2\frac{A}{B}$$ $$ A=(PPV\cdot TPR)\enspace \enspace, \enspace \enspace B=(PPV+TPR) $$ $$ \frac{\delta_A}{|A|}=\frac{\delta_{PPV}}{|PPV|} + \frac{\delta_{TPR}}{|TPR|}\Rightarrow \delta_A=|A|\left(\frac{\delta_{PPV}}{|PPV|} + \frac{\delta_{TPR}}{|TPR|}\right) $$ $$ \delta_B = \delta_{PPV} + \delta_{TPR} $$

Then we also have to combine the uncertainties for the division of $A$ by $B$:

$$ \delta_{\frac{A}{B}} = \left|\frac{A}{B}\right|\left(\frac{\delta_A}{|A|}+\frac{\delta_B}{|B|}\right) $$

This yields our final answer because we know that:

$$ F_1 \pm \delta_{F_1} \equiv 2\left(\frac{A}{B} \pm \delta_\frac{A}{B}\right) $$

Now you can see whether your $F_1$ scores overlap given some confidence interval! You could also probably see how much they overlap in terms of a ratio, but it'd probably be more understandable if you just stuck with a binary yes/no overlap.

Answer 2

This is my simple approach. We can approximate $F_{1\text{ Score}}$ and its uncertainty by K-Fold cross validation. We do the same with other model and use T-test to compare their results ($F_{1 \text{ Score}}$). It would be advisable to maintain the same partitions.

For the second question (number of test samples), I would use this formula from confidence interval (when you use the same partitions for each model), $$ N(\text{diff}, t_{score}) \geq \left( \frac{t_{score} \sigma}{\text{diff}} \right)^{2} $$ where diff is the preferred difference between the two means and $\sigma$ is the S.D. of the difference of the score for each partition of the two model. We multiply the original size of test set with N and that will be the required number of test samples. (This is just a guess and assuming you maintain the same size of training set or else this might be give good upper-bound of minimum test sample).

P.S.

1. Cross Validation works because it sums the results of all possible models, tests sets and training sets. Remember that there could be more test sets in the future and you might be even retraining your chosen model. Even now you'll be retraining your chosen model with test set to exploit a bigger data. So you need a model that wins if you consider all possible situations.
2. Most people are fooled by randomness.

---