I have built a network for the classification of three classes. The network consists of a CNN followed by two fully-connected layers. The CNN consists of convolutional layers, followed by batch normalization, a RELU activation, max pooling and drop out. The three classes are imbalanced (as can be seen in the confusion matrix below). I have optimized the parameters of the network to maximize AUC.

I'm calculating the AUC using macro- and micro-averaging. As can be seen in the ROC plot, the AUC is not that bad. On the other hand, the confusion matrix looks pretty bad, especially the first (low) class is badly predicted. The network tends to predict the majority class. As output of the network I'm getting a probability for each class. Then, I'm just taking the class according to the maximum probability for creating the confusion matrix.

I have tried to use balanced class weights while training the network (in the fit method of Keras). This helped that the network also predicts more often the minority class(es) but on the other hand the AUC was decreasing.

Is there a way to infer probability thresholds from the ROC plot? I think for two classes the optimal probability threshold can be inferred from the ROC plot by taking the max(TPR - FPR) but here I have three classes... Or is there another method?

enter image description here

enter image description here


Disclaimer: This answer describes the thoughts I had about this problem, I don't offer any guarantee about their validity so use at your own risks ;)

There are two distinct parts in this problem:

  • Finding an optimal threshold over three classes
  • Improving performance with respect to the minority class

First part: I might be wrong but as far as I know there's no way to select a particular probability threshold when there are three classes. And if there were a way, it probably wouldn't be from the ROC plot since the curves for the different plots are independent, so picking a point on the curve would correspond to different thresholds for different classes and I don't see how this would be manageable. The only way that I know is what you did: label with the class which has the maximum probability.

Second part: if you want to force the model to take care of all the classes, you could also try optimizing with macro-AUC, but that would probably lead to the same problem of decreasing the micro-AUC since there would be more errors of true neutral or high predicted as low.

The way I see it, the three way model doesn't work well: the imbalance of the minority class is not that bad, it's only 2 or 3 times less than the other classes. So the fact that the model almost completely dismisses this class is a bit strange. I also see that instances of the true class low are almost as often predicted as class high than class neutral, even though I would expect that the vast majority of errors on class low to be predicted as neutral.

So my vague intuition is that maybe the system could be designed in a way which avoids the three-way problem for the model. I can think of two options in this perspective:

  • A two-steps system where the first model classifies between low and neutral+high then the second one between neutral and high (or first with low+neutral vs. high and then low vs. neutral). This way each model is binary classification so you have more control over the thresholds at each step. Normally this is not recommended in classification, but here I assume that the classes are not truly categorical.
  • Pushing the same idea further: there seems to be an order between the classes low < neutral < high, so maybe it would be possible to treat the task as a regression problem. This might help the model avoiding these "big" errors between low and high. Importantly you would also have flexibility with the predicted values: there would be two thresholds to determine and these could be found to optimize any appropriate evaluation measure.
  • $\begingroup$ Your second idea about regression sounds interesting. You are right, there is an ordereing of the classes. So I could assign low = 0, neutral = 1 and high = 2. Furhter, I could then discretize the regresison predictions into classes and also tune the thresholds. The big problem I see here is that I don't know if the difference between the classes is 1. That means it could also be low = 0, neutral = 5 and high = 100. I don't know if there is a linear relationship. How would you tackle this problem? $\endgroup$
    – machinery
    Dec 23 '20 at 17:04
  • $\begingroup$ @machinery I agree that using 0=low, 1=neutral and 2=high is a simplification which could cause some bias, but probably not more than the separation into 3 categorical classes. Is there any kind of numerical value on which the classes are based? If yes that would be the ideal value to use as a regression target. $\endgroup$
    – Erwan
    Dec 23 '20 at 19:25
  • $\begingroup$ The classes are not based on numerical values but users are directly choosing one of the classes. So the grounth truth is built by users selecting one of the three classes. It is about emotions. low intense emotion, neutral (medium) insense and high insense emotion. I don't know if discrete values can be associated with this. $\endgroup$
    – machinery
    Dec 28 '20 at 16:43
  • $\begingroup$ @machinery in this case I would say it's worth a try, keeping in mind that it's a simplification. Anyway in the first place it's also a simplification to categorize emotions as three classes: why not a value from 1 to 10, why not a multidimensional vector for various types of emotions, etc. But you're right to be suspicious here, it would be an assumption on the data. In particular we don't know if the classes should follow a linear trend like 1,2,3, an exponential trend like 1,10,100, or anything else. This would an empirical approach, it becomes justified if it actually works. $\endgroup$
    – Erwan
    Dec 29 '20 at 0:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.