5
$\begingroup$

I'm looking for a metric that can be used to quantify how imbalanced the labels are in a dataset.

I'm not looking for a strategy to solve the imbalance problem, I just want to present how imbalanced my dataset is. I've computed the ratio of the most frequent and least frequent labels which is probably an ok way of doing it but I'm sure there's a more robust way?

Improve this question
$\endgroup$
4
  • $\begingroup$ Are you looking for sensitivity/specificity/precision and recall? Those are fairly standard. $\endgroup$
    – PSub
    Feb 25 at 5:11
  • $\begingroup$ No, I just want a metric that indicates the degree of imbalance, not a loss or accuracy. Some authors will say the ratio of the majority to minority class is 100 or 10000 etc. $\endgroup$
    – David Waterworth
    Feb 25 at 5:13
  • 2
    $\begingroup$ I see, so you mean something like $\frac{C_m}{C_i}$ where $C_m$ is the class size of the class that contains the most examples? This is in fact the weight used in weighted cross entropy for imbalanced classes. $\endgroup$
    – PSub
    Feb 25 at 5:19
  • $\begingroup$ Yes, I read a paper that did something like that but they applied over all labels to get a single figure. I just found another paper talking about multi-label classification which mentions MeanIR and IRLbl so I think what I read is related to that for multi-class classification. I may even be mistaken and what I read was specifically for multi-label. $\endgroup$
    – David Waterworth
    Feb 25 at 5:42
4
$\begingroup$

You are looking for Entropy. The higher the entropy, the more imbalanced it is. You can use this function for calculating it.

Improve this answer
$\endgroup$
4
  • $\begingroup$ So just to be clear you use the first version entropy(pk) where pk are the frequencies of each class - if my data is distributed Ci=[100,10,1] then pk would be [100/111,10/111,1/111]? (I understand the function accepts unnormalised data) $\endgroup$
    – David Waterworth
    Feb 25 at 21:10
  • $\begingroup$ Yes, you are correct. $\endgroup$
    – Abhishek Verma
    Feb 25 at 21:45
  • $\begingroup$ "The higher the entropy, the more imbalanced it is" yes, but the opposite. $\endgroup$
    – Valentas
    Mar 2 at 21:13
  • $\begingroup$ a measure of the randomness in the information being processed. The higher the entropy, the harder it is to draw any conclusions from that information. $\endgroup$
    – Abhishek Verma
    Mar 3 at 5:55
2
$\begingroup$

A very simple measure of imbalance would be the standard deviation of the classes proportions.

  • Since it's based on proportions one can compare the imbalance between different datasets
  • This takes into account all the classes, so if there are many classes it would give a different value depending on whether there are many small and many large classes (higher imbalance overall) or if there is only one outlier class (lower imbalance overall).
Improve this answer
$\endgroup$
2
$\begingroup$

I'd recommend looking at the Gini index as a measure of the inequality in the class sizes. Unlike entropy or standard deviation, Gini index is explicitly designed to capture the amount of inequality in a distribution.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.