1
$\begingroup$

From reading, I understand that when we have fewer positive class labels, it is better to use precision or recall as the evaluation metric. Which metric should I use when we have fewer negative samples?

I'm looking for an approach other than switching the labels.

Problem setting: I'm developing parametrized fragility functions for predicting damage to a structure (for example trees). An example of fragiltiy function is here The fragility function will estimate the probability of exceeding a damage state given some parameters (say wind load). The damage state can be expressed in terms of damage ratio (0-1 with 1 being fully damaged). Now, we are interested in estimating the probability of exceeding a damage ratio given features. To elaborate, the probability of any damage would be P(Damage_ratio>0.0|features). Logistic regression can be used to learn this curve from data after categorizing 0-1 continuous damage ratio to damaged (- class)/no damage (+ class) for a particular threshold. Now, as we move from the threshold from 0 to 1, the dataset transforms from imbalanced data dominated by damaged cases to a balanced state and then finally to another imbalanced data dominated by non-damage case.

Now, when learning the model, 'AUR-ROC' performs really well when the data is balanced. Precision performs well when the data is imbalanced with more no-damage case (P(Damage_ratio>0.1|features)). These metrics don't do well for the case with few negative case (P(Damage_ratio>0.9|features)). I tried switching the label with very limited success. Are there any other 'metrics' that perform well in an imbalanced data setting?

$\endgroup$
3
  • $\begingroup$ Why do you need the minority class to be called "negative"? There might be a good reason but you should mention it imho. What is the task? $\endgroup$
    – Erwan
    Nov 10, 2020 at 21:46
  • $\begingroup$ Thanks for adding details, it looks to me like the problem is not with the evaluation measure: the smaller the minority class, the harder it is for the classifier, so it's normal that the performance decreases. Changing the evaluation measure would be like hoping that changing the thermometer will make the fever disappear ;) However have you considered treating the problem as a regression task, i.e. predicting directly the "damage ratio" value instead of a class? $\endgroup$
    – Erwan
    Nov 10, 2020 at 23:42
  • $\begingroup$ :) Thank @Erwan. I agree with you; for extreme damage ratios, I have only a few samples to learn. My intention was to get the best result possible given the constraints and one way we could do that is to use appropriate metrics for training (AUC-ROC vs AUC-PR). Yes, I am developing a regression model in addition to the classification model. Thanks a lot. $\endgroup$
    – PPR
    Nov 11, 2020 at 0:04

2 Answers 2

1
$\begingroup$

The names of the classes don't matter, you might as well call them class A and class B. In binary classification the typical choice is to evaluate using precision, recall and F1-score. There are other options, but that depends on the task.

Assuming you choose F1-score, the choice of which class you select as the "positive" class for evaluation also depends on the task. Usually it's recommended to use the minority class because it's the most challenging one for the classifier.

The only problem here is the possible confusion of calling a class "negative" and calculating F1-score using it as the "positive" class, but that's just a naming issue. You could easily add this point to the explanations, or avoid any confusion by calling your classes A and B for instance.

$\endgroup$
0
$\begingroup$

The most recommended would be precision, recall and F1-Score, but there are others like F-Beta score or other threshold metrics.

In any case, based on my experience, the choice of the metric depends on your use case and the conditions under which the classifier will be deployed (eg. if your production data does have a different ratio observed between the classes, your metrics performance can be misleading).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.