# Hyperparameter tuning in multiclass classification problem: which scoring metric?

I'm working with an imbalanced multi-class dataset. I try to tune the parameters of a DecisionTreeClassifier, RandomForestClassifier and a GradientBoostingClassifier using a randomized search and a bayesian search.

For now, I used just accuracy for the scoring which is not really applicable for assessing my models performance (which I'm not doing). Is it also not suitable for parameter tuning?

I found that for example recall_micro and recall_weighted yield the same results as accuracy. This should be the same for other metrics like f1_micro.

So my question is: Is the scoring relevant for tuning? I see that recall_macro leads to lower results since it doesn't take the number of samples per class into account. So which metric should I use?

• Yes, but for deciding purposes. The score helps you when to stop training. – Media Apr 26 '18 at 13:25
• So if I just use a maximum number of iterations to decide when to stop tuning, its irelevant if I use accuracy or recall? – Christian Apr 26 '18 at 13:38
• No, based on accuracy or recall you have to decide whether you stop your training or not, whether increase the number of iterations or not. – Media Apr 26 '18 at 15:16
• I think I dont fully understand your point. What does the scoring used in parameter tuning has to do with stopping the training? If I use a randomized parameter search for example the scoring metric is only used to rank the models and they have the same rank using accuracy or recall_weighted and recall_micro. – Christian Apr 26 '18 at 15:20
• Suppose that you have unbalanced data-set. 99% of your training data has label 0 and 1% of your data has label 1. In this case if your model always outputs 0, you will have a model with 99% accuracy and you won't train in anymore. If you use F1 score, your evaluation method tells you that you are in a wrong path and you continue training. :) – Media Apr 26 '18 at 15:24

A simple solution is to set importance weight in front of each class inversely proportional to the train set relative frequency of the class like $$\frac{1}{freq}$$ or $$e^{-freq}$$. The choice of the right formula depends on how much importance you would give to less frequent classes
e.g. $$e^{-freq}$$ give more importance to less frequent classes than $$\frac{1}{freq}$$