1
$\begingroup$

after looking a lot in the literature there is really a lot of how to work with imbalanced dataset but so far I can not find a

definition of a imbalanced metric that quantifies how much imbalanced is dataset A compared to dataset B.

I have even tried to define one such metric myself but I fail to capture with a single metric the following two cases:

  1. One class has all the measurements
  2. Measurements are more balanced but one or two classes have very low measurement or even zero.

Can you give me some more information from the literature that perhaps I was not able to find out or at least try to discuss the issue of definition of imbalanceness .

Thanks a lot Alex

$\endgroup$
1
$\begingroup$

I think "balanced" can have different meanings in different contexts, but typically a "balanced" dataset is one in which the class labels are uniformly distributed. Of course, an "imbalanced" dataset is one in which the distribution of class labels is not uniform.

I'm not aware of discussions about balance metrics in the literature. One straightforward option is to measure the KL-divergence between the actual distribution of the class labels and a uniform distribution over the class labels. A KL-divergence of zero would indicate a perfectly balanced dataset. The KL-divergence will grow to infinity as the actual distribution differs from the perfectly-balanced distribution. This metric would allow you to compare the degree of imbalance between two datasets.

Unfortunately, KL-divergence is pretty fragile. It's technically undefined when one of the classes has zero examples. In practice you could treat such datasets as if they're infinitely imbalanced. This is what scipy.stats.entropy does.

However, this treatment has one undesirable side-effect. Any dataset in which there are zero examples of a class gets the same "imbalance score" (infinity). This doesn't match our intuition about imbalance. Suppose we have two datasets with examples from the classes "dog", "cat", and "horse". Let's say dataset A has 19 dogs, 1 cat, and 0 horses. Dataset B has 10 dogs, 10 cats, and 0 horses. Ideally our metric would say that Dataset A is more imbalanced than Dataset B, but KL-divergence will give the same result for both datasets.

We can correct this problem by adding a tiny epsilon to each of the class distributions before calculating KL-divergence. This both solves the problem of being undefined when a class has zero examples, and it also corrects for the above scenario where two different datasets get the same imbalance score.

$\endgroup$
  • $\begingroup$ Thanks a lot. Very detailed reply! I appreciate. KL divergence looks to be a way forward and correct for classes with zero samples should be easy (just adding 1+2 elements in two all classes should fix that). I am mostly worried now on how much you think KL divergence can be used to give that rating and then one can even use that rating for doing binary labeling (imbalanced or not). I am thinking that KL divergence will very fast penalize also datasets that are somehow balanced. $\endgroup$ – Alex P Dec 12 '19 at 6:02
  • $\begingroup$ Hi Alex, sorry to be so late with my reply. I think the only way to use KL-divergence for binary labeling like you describe is to threshold the divergence. Setting the threshold will probably be an empirical matter, since a reasonable threshold will depend on the number of classes and of course your subjective evaluation of how much imbalance is permissible. $\endgroup$ – zachdj Dec 16 '19 at 22:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.