I have a multi-class classification task where the organizers said that the final results will be using the Accuracy measure.

The provided data is unbalanced, and I don't have an idea about the test set (is it balanced or not), but I think it will be balanced since they use accuracy.

Anyway ..

My question: Is it a good idea to tune my system using F1-macro rather than Accuracy? since the training data is unbalanced.

or it's better to use the Accuracy?

  • $\begingroup$ It's a better approach to use F1. Will the organizers evaluate all your process or only your results? I'd tune my model with F1 and then deliver it with the accuracy $\endgroup$
    – ignatius
    Dec 5 '18 at 15:48
  • 1
    $\begingroup$ Also, they might want to evaluate how you approach the problem... The reason behind giving an unbalanced data-set and accuracy as metric might be to check whether you notice the problems with that and how you face it, for example balancing your data in some way $\endgroup$
    – ignatius
    Dec 5 '18 at 15:50
  • $\begingroup$ only my results .. and thanx for the suggestion $\endgroup$
    – Minions
    Dec 5 '18 at 16:18
  • 1
    $\begingroup$ Well, so nothing prevents you from tuning the model with a metric of your choice. Good luck! $\endgroup$
    – ignatius
    Dec 5 '18 at 16:20

using accuracy for unbalance data means that correct classification for the most populous class members is more important than others. If the importance of correct classifying for all data records is equal in your problem accuracy is one of the worst choices.

There are some other good choices beside F1-macro which can be more helpful. Some of these metrics are as follows:

"Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN",
"MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC",
"AUCI", "G", "DP", "DPI", "GI"


If you use python, PyCM module can help you to find out these metrics.

Here is a simple code to get the recommended parameters from this module:

>>> from pycm import *

>>> cm = ConfusionMatrix(matrix={"Class1": {"Class1": 1, "Class2":2}, "Class2": {"Class1": 0, "Class2": 5}})  

>>> print(cm.recommended_list)
["Kappa", "SOA1(Landis & Koch)", "SOA2(Fleiss)", "SOA3(Altman)", "SOA4(Cicchetti)", "CEN", "MCEN", "MCC", "J", "Overall J", "Overall MCC", "Overall CEN", "Overall MCEN", "AUC", "AUCI", "G", "DP", "DPI", "GI"]

After that, each of these parameters you want to use as the loss function can be used as follows:

>>> y_pred = model.predict      #the prediction of the implemented model

>>> y_actu = data.target        #data labels

>>> cm = ConfusionMatrix(y_actu, y_pred)

>>> loss = cm.Kappa             #or any other parameter (Example: cm.SOA1)

You should definitely use macro-average F1 as the accuracy could be highly biased by the majority class. The F1 makes an harmonic mean of recall and precision, giving a trade-off measure considering what has been correctly predicted and what not.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.