5
$\begingroup$

I'm working with an imbalanced multi-class dataset. I try to tune the parameters of a DecisionTreeClassifier, RandomForestClassifier and a GradientBoostingClassifier using a randomized search and a bayesian search.

For now, I used just accuracy for the scoring which is not really applicable for assessing my models performance (which I'm not doing). Is it also not suitable for parameter tuning?

I found that for example recall_micro and recall_weighted yield the same results as accuracy. This should be the same for other metrics like f1_micro.

So my question is: Is the scoring relevant for tuning? I see that recall_macro leads to lower results since it doesn't take the number of samples per class into account. So which metric should I use?

Improve this question
$\endgroup$
11
  • $\begingroup$ Yes, but for deciding purposes. The score helps you when to stop training. $\endgroup$ – Media Apr 26 '18 at 13:25
  • $\begingroup$ So if I just use a maximum number of iterations to decide when to stop tuning, its irelevant if I use accuracy or recall? $\endgroup$ – Christian Apr 26 '18 at 13:38
  • $\begingroup$ No, based on accuracy or recall you have to decide whether you stop your training or not, whether increase the number of iterations or not. $\endgroup$ – Media Apr 26 '18 at 15:16
  • $\begingroup$ I think I dont fully understand your point. What does the scoring used in parameter tuning has to do with stopping the training? If I use a randomized parameter search for example the scoring metric is only used to rank the models and they have the same rank using accuracy or recall_weighted and recall_micro. $\endgroup$ – Christian Apr 26 '18 at 15:20
  • $\begingroup$ Suppose that you have unbalanced data-set. 99% of your training data has label 0 and 1% of your data has label 1. In this case if your model always outputs 0, you will have a model with 99% accuracy and you won't train in anymore. If you use F1 score, your evaluation method tells you that you are in a wrong path and you continue training. :) $\endgroup$ – Media Apr 26 '18 at 15:24
0
$\begingroup$

You should use the same metric to evaluate and to tune the classifiers. If you wull evaluate the final classifier using accuracy, then you must use accuracy to tune the hyper parameters. If you think you should use macro-averaged F1 as the final evaluation of the classifier, use it also to tune them.

On a side, for multiclass problems I have not yet heard any convincing argument not to use accuracy, but that is just me.

Improve this answer
$\endgroup$
0
$\begingroup$

If your dataset is imbalance then you can calculate the kappa score.

Improve this answer
$\endgroup$
0
$\begingroup$

A simple solution is to set importance weight in front of each class inversely proportional to the train set relative frequency of the class like $\frac{1}{freq}$ or $e^{-freq}$. The choice of the right formula depends on how much importance you would give to less frequent classes
e.g. $e^{-freq}$ give more importance to less frequent classes than $\frac{1}{freq}$

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.