1
$\begingroup$

In one of the research papers on fake news detection, the authors collected a fake news binary dataset (fake vs. real news) consists of 16,817 real articles and 5,323 fake ones.

The authors presented the results using accuracy, precision, recall and F1 without specifying which kind of averaging they applied on the F1 metric (macro, micro, weighted, etc.).

Here are the results:

enter image description here

If you can notice for the last system, the accuracy value is 0.689 and the F1 value is 0.717 which is higher than the accuracy.

Thus, give the imbalanced status of the dataset, is it possible that the authors averaged the classes in the F1 metric using macro way?

For me, this "impossible" to happen, and I argue that probably they used weighed F1 score.

| improve this question | |
$\endgroup$
1
$\begingroup$

They calculated the "standard" F1 score as defined for binary classification tasks:

precision = 0.656
recall = 0.792
f1 = 2 * (precision * recall) / (precision + recall)
f1

gives

0.7176132596685083

Other versions of the F1 score are used for multiple classes as you can see here under "Extension to multi-class classification":

The F-score is also used for evaluating classification problems with more than two classes (Multiclass classification). In this setup, the final score is obtained by micro-averaging (biased by class frequency) or macro-averaging (taking all classes as equally important). For macro-averaging, two different formulas have been used by applicants: the F-score of (arithmetic) class-wise precision and recall means or the arithmetic mean of class-wise F-scores, where the latter exhibits more desirable properties.

Alternatively, see here for the scikit learn implementation of the F1 score and its parameter description.

| improve this answer | |
$\endgroup$
  • $\begingroup$ what do you mean by standard F1? the issue is also regarding precision & recall. I mean, to extract precision & recall you have to do averaging for the classes. $\endgroup$ – Ghanem Feb 18 at 17:51
  • $\begingroup$ @Ghanem I mean the F1 score as I defined in the code of my answer ($F1 = 2\times(precision\times recall) / (precision + recall)$). That is the standard definition for binary classification tasks. Precision and recall do not require averaging in a binary task either: $precision = TP / (TP + FP)$ and $recall = TP / P$ where $TP$ stands for true positives, $FP$ for false positives and $P$ for all positives. As for the F1 score you only need averages or weights for multi-class-tasks - not for binary. $\endgroup$ – Sammy Feb 18 at 18:19
  • $\begingroup$ Have a look on this code: from sklearn.metrics import f1_score y_true = [0, 1, 0, 1, 0, 0, 1, 0, 1] predicted = [1, 1, 1, 1, 0, 0, 1, 0, 1] print(f1_score(y_true, predicted, average='binary')) print(f1_score(y_true, predicted, average='weighted')) print(f1_score(y_true, predicted, average='macro')) $\endgroup$ – Ghanem Feb 18 at 18:25
  • $\begingroup$ @Ghanem have a look at the scikit learn documentation and the average parameter description: "This parameter is required for multiclass/multilabel targets. If None, the scores for each class are returned. Otherwise, this determines the type of averaging performed on the data". The default is average="binary" which is what I referred to as "standard". $\endgroup$ – Sammy Feb 18 at 18:28
  • $\begingroup$ Ah ok, you mean standard=binary. This is different. Anyway macro with binary dataset gives different result comparing to binary. $\endgroup$ – Ghanem Feb 18 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.