5
$\begingroup$

I'm working with an imbalanced multi-class dataset. I try to tune the parameters of a DecisionTreeClassifier, RandomForestClassifier and a GradientBoostingClassifier using a randomized search and a bayesian search.

For now, I used just accuracy for the scoring which is not really applicable for assessing my models performance (which I'm not doing). Is it also not suitable for parameter tuning?

I found that for example recall_micro and recall_weighted yield the same results as accuracy. This should be the same for other metrics like f1_micro.

So my question is: Is the scoring relevant for tuning? I see that recall_macro leads to lower results since it doesn't take the number of samples per class into account. So which metric should I use?

$\endgroup$
11
  • $\begingroup$ Yes, but for deciding purposes. The score helps you when to stop training. $\endgroup$ Apr 26, 2018 at 13:25
  • $\begingroup$ So if I just use a maximum number of iterations to decide when to stop tuning, its irelevant if I use accuracy or recall? $\endgroup$
    – Christian
    Apr 26, 2018 at 13:38
  • $\begingroup$ No, based on accuracy or recall you have to decide whether you stop your training or not, whether increase the number of iterations or not. $\endgroup$ Apr 26, 2018 at 15:16
  • $\begingroup$ I think I dont fully understand your point. What does the scoring used in parameter tuning has to do with stopping the training? If I use a randomized parameter search for example the scoring metric is only used to rank the models and they have the same rank using accuracy or recall_weighted and recall_micro. $\endgroup$
    – Christian
    Apr 26, 2018 at 15:20
  • $\begingroup$ Suppose that you have unbalanced data-set. 99% of your training data has label 0 and 1% of your data has label 1. In this case if your model always outputs 0, you will have a model with 99% accuracy and you won't train in anymore. If you use F1 score, your evaluation method tells you that you are in a wrong path and you continue training. :) $\endgroup$ Apr 26, 2018 at 15:24

4 Answers 4

1
$\begingroup$

You should use the same metric to evaluate and to tune the classifiers. If you wull evaluate the final classifier using accuracy, then you must use accuracy to tune the hyper parameters. If you think you should use macro-averaged F1 as the final evaluation of the classifier, use it also to tune them.

On a side, for multiclass problems I have not yet heard any convincing argument not to use accuracy, but that is just me.

$\endgroup$
2
1
$\begingroup$

The evaluation metric depends on the goals of the project. Which outcomes are better or which outcomes worse? Some projects value precision over recall and other projects value recall over precision.

After you have clarity on project goals, pick a single metric to provide a consistent scorecard when comparing different algorithms and hyperparameters combinations. One common evaluation metric for multi-class classification is F-score. F-score has a β hyperparameter which weights recall and precision differently. You will have to choose between micro-averaging (biased by class frequency) or macro-averaging (taking all classes as equally important). For macro-averaging, two different formulas can be used:

  1. The F-score of (arithmetic) class-wise precision and recall means.

  2. The arithmetic mean of class-wise F-scores (often more desirable).

$\endgroup$
0
$\begingroup$

If your dataset is imbalance then you can calculate the kappa score.

$\endgroup$
0
$\begingroup$

A simple solution is to set importance weight in front of each class inversely proportional to the train set relative frequency of the class like $\frac{1}{freq}$ or $e^{-freq}$. The choice of the right formula depends on how much importance you would give to less frequent classes
e.g. $e^{-freq}$ give more importance to less frequent classes than $\frac{1}{freq}$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.