5
$\begingroup$

I have two multiclass classification models for making predictions (number of classes is three to be precise). One is Keras neural network, other is Gradient Boosted Classifier from Scikit Learn library.

I have noticed that after training on same data GBC has higher accuracy score, while Keras model has higher F1 score. Which model should I use for making predictions on future data? Which metric is more important?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ This is more of a business question, what business problem are you trying to solve? If you are working with highly imbalanced problem, accuracy can be misleading. $\endgroup$
    – Akavall
    Dec 23 '19 at 17:53
  • $\begingroup$ @akavall, forgot to mention that my data is not imbalanced (one of the classes is, but both models still predict that class quite consistently) $\endgroup$
    – Ach113
    Dec 23 '19 at 18:18
9
$\begingroup$

Well it is highly dependent on your use cases and how your data is distributed.

Let me breakdown the pros and cons in practice :

F1-Score

Pros :

  • Takes into account how the data is distributed. Useful when you have data with imbalance classes.

Cons :

  • Less interpretable. Precision and recall is more interpretable than f1-score, since it measures the type-1 error and type-2 error. However, f1-score measures the trade-off between this two.
  • When positive class is minority class, the score is quite sensitive when there is switching where the ground truth is positive.

Accuracy

Pros :

  • Easy to understand.

Cons :

  • It does not take into account how the data is distributed (example case below). This error could be crucial and might lead to incorrect conclusion.

Here is an example depicting cons of accuracy. Without loss of generality consider a binary classification task. Imagine having a data which of 100 samples with 90 negative sample and 10 positive sample. Suppose you have a classifier that predicts all negative. You will have an accuracy of 90%, but let's consider the f1 score, you will actually get 0 because your recall (which is a component of f1 score) is 0.

In practice, for multi-class classification model (which is your use-cases) accuracy is mostly favored. f1 is usually used for multi-label or binary label where the classes are highly unbalanced.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I see, in my case data is quite balanced (not sure about exact percentages but its definitely not like in your example), GBC gave 86% accuracy with 82 F1 score, Keras DNN gave 84% accuracy with 85 F1. And I dont really know which one to use. $\endgroup$
    – Ach113
    Dec 23 '19 at 18:17
  • 1
    $\begingroup$ I think I would say accuracy might be what you are looking for. Using f1 for multiclass classification is quite rare unless your imbalance is quite extreme and you wish to capture it. You can also add log_loss as another metric for your judgement. Metric is only to give you information, how you use the information is up to you. $\endgroup$
    – Yohanes Alfredo
    Dec 24 '19 at 3:40
  • $\begingroup$ I would say that, for a generic case also in multiclass classification, you should look for a robust metric which takes into account several decision thresholds, i.e. ROC AUC: scikit-learn.org/stable/modules/generated/… $\endgroup$
    – German C M
    Dec 24 '19 at 10:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.