3
$\begingroup$

One of the metrics that is widely used in binary classification is the F1 score:

$F_1 = 2\cdot \frac{recall \cdot precision}{recall+precision}$

The problem of the F1-score is that it is not differentiable and so we cannot use it as a loss function to compute gradients and update the weights when training the model. The F1-score needs binary predictions (0/1) to be measured.

I am seeing it a lot. Let's say I am using per example a Linear regression or a gradient boosting.

Is there any way that it can be minimized directly?

$\endgroup$
3
4
$\begingroup$

Yes there is, let's take $F_1$ score base definition, with : $$ F_1 = 2 \times \frac{precision \times recall} {precision + recall} \\ F_1 = \frac{2 \times TP} {2 \times TP + FP + FN} $$ And this is the same as the Sørensen-Dice coefficient, also known as Dice coefficient or Bray-Curtis distance. This is a statistical indicator that measures the similarity of two samples :

$$ Dice(X,Y) = \frac{2|X \cap Y|}{|X| + |Y|}$$

Concerning the implementation of this loss, we can approximate $|X \cap Y|$ as the sum of the matrix obtained using Hadamard product ($\odot$, also known as the element-wise product) between the ground truth ($y$) and the prediction ($\hat{y}$). We can then define $L_{Dice}$ as follows:

\begin{align*} L_{Dice} &= 1 - Dice \\ L_{Dice}\left(y, \hat{y}\right) &= 1 - \frac{ 2\sum y \odot \hat{y}} {\sum y + \sum \hat{y}} \end{align*}

You will often find this loss in the context of segmentation problems, as well as others quite close, such as the Jaccard index (IoU).

$\endgroup$
5
  • 1
    $\begingroup$ Isn't $\sum y \odot \hat{y}$ the same as the dot product of $y$ and $\hat{y}$, since both are one-dimensional vectors? $\endgroup$ – Milos Jun 13 at 18:36
  • $\begingroup$ I like it. What are the gradient and hessian? We take derivative w.r.t. $\hat{y}$, right? $\endgroup$ – Pavel Komarov Jul 14 at 21:33
  • $\begingroup$ I'm trying to implement this as an objective for XGBClassifier. xgboost.readthedocs.io/en/latest/python/python_api.html $\endgroup$ – Pavel Komarov Jul 14 at 21:43
  • $\begingroup$ If I consider each $y_i$, $\hat{y_i}$ to be independent, then my loss becomes $L_i = 1 - \frac{2 y_i \hat{y_i}}{y_i + \hat{y_i}}$. If I use a sigmoid for $\hat{y_i}$ to let the model predict a logit, which we squash to a probability, then $L_i = 1 - \frac{2 y_i \sigma(\hat{y_i})}{y_i + \sigma(\hat{y_i})}$. Then first and second derivative w.r.t. $\hat{y_i}$ can be found: wolframalpha.com/input/…, wolframalpha.com/input/… $\endgroup$ – Pavel Komarov Jul 16 at 22:01
  • $\begingroup$ I get better results using the cross entropy loss, though: $L(y,\hat{y}) = -\frac{1}{N} \sum_{i} y_i log(\sigma(\hat{y_i})) + (1-y_i) log (1-\sigma(\hat{y_i}))$, where $\sigma(x) = \frac{1}{1 + e^{-x}}$ -> $L_i = - (y_i log(\sigma(\hat{y_i})) - (1-y_i) log(1-\sigma(\hat{y_i})))$ -> $\frac{\partial}{\partial \hat{y_i}} L_i = -y_i + \sigma(\hat{y_i}) $ and $\frac{\partial^2}{\partial \hat{y_i}^2} L_i = \sigma(\hat{y_i})(1 - \sigma(\hat{y_i}))$ $\endgroup$ – Pavel Komarov Jul 16 at 22:06
2
$\begingroup$

Following on Thomas, on the relation between the Bray-Curtis distance and the F1 score and the calculation of the first and second-order derivatives: If one defines the Bray Curtis distance between vector X and Vector Y as: $\sum |X_i-Y_i| \over {\sum (X_i+Y_i)}$, than the first derivative to $x$ is $d \over (dx)$ $|x - y| \over {(x + y)}$ = $2y(x - y) \over{\big((x + y)^2(|x - y|)\big)}$ and the second derivative is ${d^2 \over{dx^2}} {|x - y| \over(x + y)} = {-4y*(x - y) \over\big((x + y)^3*(|x - y|)\big)}$

$\endgroup$
1
  • $\begingroup$ How can I use this? As $x$, or as I might call it $\hat{y}$ approaches $y$, the gradient and hessian explode, because the denominator goes to 0. Seems like the optimum must be an unstable one. $\endgroup$ – Pavel Komarov Jul 16 at 22:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.