How can I say if I have a class imbalance issue in my data?

Question

Assuming I have this dataset:

Label --- %Total
0 -------- 18.53%
1 -------- 8.18%
2 -------- 26.22%
3 -------- 16.46%
4 -------- 8.62%
5 -------- 9.58%
6 -------- 5.88%
7 -------- 6.53%

I could say I have a class imbalance problem ? Is it mandatory in this case to fix the problem trying to use all the various techniques (resampling, data augmentation, change perf metric etc...) ?

Is there a mathematical formula to get the grade of imbalance severity or something like that to understand if there is a class imbalance problem ?

I think we have to evaluate case by case, the techniques to avoid imbalanced data could even not work at all, there isn't a general rule. Any ideas?

Answer 1

'Imbalance problem' is a mix up of several loosely related issues, mainly these two:

1. It's hard to generalize when there's too few of a certain class' samples, especially with lots of dimensions. However, methods like resampling won't help much in this case: in an oversimplified way, that means trying to combat model variance by shifting its bias. There's little you can do aside from gathering more data unless, perhaps, you are only interested in certain class-specific metrics of those few rare classes. Your class distribution does not seem that bad - your model will generalize alright with enough samples regardless of the class ratio.

2. Logistic functions underestimating the rare cases' probability. That's basically just bias, resampling / reweighing / threshold selection have mostly the same effect. The latter is the easiest as it does not require retraining, however this is, strictly speaking, a decision making part, which should not be mixed with evaluation stage (there could be more than one decision threshold for different actions etc).

So, the 'ideal' way would be: don't resample at all, evaluate using 'proper' (class independent and threshold independent) metrics, such as logloss, and thus work directly with scores/probabilities (calibrate if needed) up until the decision stage.

In DS context however, you often still need 'intuitive' metrics (based upon confusion matrix), which are threshold sensitive and often class specific. Even then, anything more complex than selecting a threshold upon the precision/recall curve is usually excessive.