10
$\begingroup$

Why is the F-measure usually used for (supervised) classification tasks, whereas the G-measure (or Fowlkes–Mallows index) is generally used for (unsupervised) clustering tasks?

The F-measure is the harmonic mean of the precision and recall.

The G-measure (or Fowlkes–Mallows index) is the geometric mean of the precision and recall.

Below is a plot of the different means.

enter image description here

F1 (harmonic) $= 2\cdot\frac{precision\cdot recall}{precision + recall}$

Geometric $= \sqrt{precision\cdot recall}$

Arithmetic $= \frac{precision + recall}{2}$

The reason I ask is that I need to decide which average to use in a NLG task, where I measured BLEU and ROUGE ( where BLEU is equivalent to precision and ROUGE to recall). How should I calculate the mean of these scores?

$\endgroup$
  • $\begingroup$ Maybe it's just how the defination goes! $\endgroup$ – Aditya Aug 12 '18 at 9:42
  • 1
    $\begingroup$ @Aditya, you are right, it was just a badly formulated questions about definition. I edited it reformulating into something more concrete. $\endgroup$ – Bruno Lubascher Aug 12 '18 at 9:57
3
$\begingroup$

The Fı-score is preferred to simple classification accuracy in order to counter the problem of imbalanced datasets; if the thing you are looking for occurs only rarely anyway then a naive classifier can always say no and appear to be working very well! A variant on Fı is Fß, where

Fß = (1+ß²) × [ (P × R) ÷ ( (ß² × P) + R ) ]

Vary ß to balance precision and recall. As to the why F or G, I believe it to be empirical - you don't say if you are classifying or clustering in your own application?

$\endgroup$
  • 3
    $\begingroup$ Thanks for the answer, but I think you miss understood my question. I don't mean to compare F1 vs. simple accuracy., instead, I mean to compare the Harmonic vs Geometric vs Arithmetic means. I am not doing traditional classification or clustering, I have a NLG task, which is measured in BLEU and ROUGE which could be averaged with one of the means, but I am not sure which to pick. $\endgroup$ – Bruno Lubascher Aug 12 '18 at 14:56
0
$\begingroup$

The scoring function is used as an objective measure of performance. The choice of scoring function itself is subjective and should reflect what you or the problem deems to be important in terms of the balance between whatever metrics you are tracking (e.g., precision & recall, or sensitivity & specificity, or BLEU & ROUGE).

Arithmetic mean, geometric mean, and harmonic mean are all special cases of the generalized means family, which means they're conceptually related. For your task, the arithmetic mean represents no preference between whether BLEU or ROUGE is higher, and where increasing one value and decreasing the other by the same amount makes no difference. The geometric and harmonic means both penalise differences between BLEU and ROUGE, with the harmonic mean being more "pessimistic" than the geometric mean. This can be seen in your plot, where the arithmetic curve sits above the geometric curve, and the harmonic curve is at the bottom. Using the generalized mean, you could have subjectively chosen any curve above the arithmetic curve, below the harmonic curve, or anywhere between these. There is no inherent reason why the harmonic mean or geometric mean is more meaningful, they just have simple formulas. Pick whatever is a closer match for how you value the trade-off between BLEU and ROUGE. Equally, you may decide you don't want to use any of these means based on the generalised mean.

New contributor
Vic is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
-3
$\begingroup$

If Precision and Recall are similar, F1 is a good single measure to compare different models.

Short and sweet :)

$\endgroup$
  • 1
    $\begingroup$ I don't see how you even attempted to answer my question... $\endgroup$ – Bruno Lubascher Feb 23 '19 at 18:22
  • $\begingroup$ short and unclear $\endgroup$ – Julio Jesus Jan 17 at 23:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.