4
$\begingroup$

I know the general expression of the F1-score:

$$F1 = \frac{precision * recall}{precision + recall}$$

And its $F_{beta}$ variants (see: https://en.wikipedia.org/wiki/F-score):

$$F_{beta} = (1+\beta^2) * \frac{precision * recall}{\beta^2 * precision + recall}$$

I was wondering what optimisation problem it solve. That is F1 is defined as the harmonic mean, and $F_{beta}$ is the harmonic mean with precision weighted by $\beta$. But I have a hard time linking this with a cost matrix.

That is with the cost matrix:

True Predicted: 0 Predicted: 1
0 a b
1 c d

In terms of a,b,c and d what is maximising $F_{beta}$ achieving ? Can the $\beta$ imbalance can be linked to imbalance between a,b,c and d?

$\endgroup$
2
  • $\begingroup$ If you want to the F-score optimizes this differentiable loss stackoverflow.com/a/65320239/12229416 $\endgroup$
    – Ggjj11
    Commented Feb 8 at 7:37
  • $\begingroup$ The question is not about how to optimise F1 score but rather asking for a justification of it being a good idea. $\endgroup$ Commented Feb 8 at 8:16

1 Answer 1

3
+200
$\begingroup$

There isn't a direct and perfect correspondence between the optimal cost-based objective and the optimal F1 score.

Here's an example to show that, though it's perhaps somewhat unsatisfactory because it essentially constructs a sufficiently imbalanced dataset to break things.

$\DeclareMathOperator{tp}{TP}\DeclareMathOperator{fp}{FP}\DeclareMathOperator{tn}{TN}\DeclareMathOperator{fn}{FN}$ I'll take as convention that the cost-based objective is $a\tn-b\fp-c\fn+d\tp$, with $a,b,c,d\geq0$, and we're seeking the maximum.

If $a,b>0$, then let $x=\max\{2c/a, 2d/b\}$, and consider the two confusion matrices

$$\begin{bmatrix}x&0\\1&0\end{bmatrix}, \begin{bmatrix}0&x\\0&1\end{bmatrix}$$

The first one has $F_1=0$ and cost-based objective $xa-c\geq c\geq0$; the second one has $F_1=2/(2+x)>0$ and objective $d-bx\leq -d \leq 0$. So the two metrics disagree about which confusion matrix is better.

(If one of $a,b$ is zero, I think this can still be salvaged, you just need to take $x$ large enough. If they're both zero then it doesn't matter what $x$ is, the example works.)

I'd be happy to hear an example that didn't rely so much on imbalance.


From a higher level, rewrite $$F_1 = \frac{2\tp}{2\tp+\fp+\fn} = \frac{1}{1+\frac{\fp+\fn}{2\tp}},$$ to see that maximizing $F_1$ is equivalent to minimizing $\frac{\fp+\fn}{\tp}$. Thinking about the case $a=0$, $b=c=d=1$, we're still looking at optimizing the ratio of errors to true positives vs optimizing the difference between errors and true positives.

(That leads to a nice 2d optimization, with feasible region a triangle, F1 level curves being lines through the origin of differing slopes, and objective level curves being parallel lines.)

$\endgroup$
1
  • $\begingroup$ Thanks for the discussion. I was toying with the objective too. I think without lack of generalisation we can assume a=1. Given the symetry of the F1 formula, I think we need b=c. So in the end the problem would only be paramatrized with the relative cost of error. Also we can use TP + FP + TN + FN = 1 to simplify the cost. However in the end there need to be a division by TP that is needed to make the F1 score appears in the cost. $\endgroup$ Commented Feb 8 at 7:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.