5
$\begingroup$

I have a set of data with a few strongly imbalanced classes, eg the smallest class is about 54 times smaller than the largest. Therefore, data augmentation in order to equalize the size of classes seems to me a bad idea (in the example above each image would have to be augmented 54 times on average). So I thought that I could do less augmentation of minority classes, and then use class weights in the loss function. Is this approach better than the mere augmentation or just the use of class weights ?

$\endgroup$
6
$\begingroup$

Is this approach better than the mere augmentation or just the use of class weights ?

Note that data augmentation is the process of changing the training samples (e.g. for images, flipping them, changing their luminosity, adding noise, etc.) and adding them back into the set. It is used for enriching the diversity of training samples, thus, in this aspect it cannot be replaced with class weighting. However, it is related to over-sampling since we can both enrich and increase the size of a class via data-augmentation. For now we are only focusing on the size increasing part, i.e. data over-sampling.

Data-oversampling and class weighting are equivalent. Copying the samples of a class 3X is equivalent to assigning a 3X weight to the class. However, the weighting is better from storage and computational point of view since it avoids working with a larger data-set.

Note that based on this equivalency we can mix and match. Increasing the size of a class 6X $\equiv$ increasing the weight 6X $\equiv$ increasing the size 3X and increasing the weight 2X.

Class weights vs over-sampling in more detail

I copied my answer from here, since the solutions are almost the same but questions are the reverse of each other.

This challenge can be tackled in two places:

  1. Data: as you mentioned, this is done by artificially increasing the number of samples for under-represented class $uc$. This produces the same effect as data-sets that are naturally balanced,

  2. Model: this is generally done by over-penalizing the miss-classification of $uc$ compared to other classes. One place for this modification is the loss function. A frequently used loss function in classification is cross entropy. It can be modified for this purpose as follows. Let $y_{ik}$ be $1$ if $k$ is the true class of data point $i$, and be $0$ otherwise, and $y'_{ik} \in (0, 1]$ be the corresponding model estimation. The original cross-entropy can be written as: $$H_y(y')=-\sum_{i}\sum_{k=1}^{K}y_{ik}log(y'_{ik})$$
    which can be weighted as $$H_y(y')=-\sum_{i}\sum_{k=1}^{K}\color{blue}{w_{k}}y_{ik}log(y'_{ik})$$ For example, by setting $w_{uc} = 10$ and $w_{k \neq uc}=1$, we are essentially telling the model that miss-classifying $1$ member from $uc$ is as punishable as miss-classifying $10$ members from other classes. This is roughly equivalent to increasing the ratio of class $uc$ $10$ times in the training set using method (1).
    As a concrete example, suppose ratio of three classes are $c_1=10\%$, $c_2=60\%$, and $c_3=30\%$, thus, we can set $w_{c_1}=6$, $w_{c_2}=1$, and $w_{c_3}=2$, which implies $w_{c_i} \times P(c_i)=w_{c_j}\times P(c_j)$ for all class pairs $c_i$ and $c_j$.

$\endgroup$
  • $\begingroup$ Thank you for your response. "Both are equivalent. Copying the samples of a class 3 times is equivalent to assigning a 3X weight to the class. However, the weighting is better from storage and computational point of view since it avoids working with a larger data-set." - I think you are talking about weighing examples, not data augumentation. I wanted to know if the hybrid approach, i.e. augmentation + class weights, in case of such a large class imbalance is better than using only augumentation or class weights. $\endgroup$ – I.D.M Mar 16 '19 at 18:03
  • 2
    $\begingroup$ Data augmentation is not equivalent to changing class weights!!!! Oversampling is equivalent. Data augmentation can reveal more of the true data generating distribution $\endgroup$ – Bashar Haddad Mar 16 '19 at 18:06
  • $\begingroup$ @BasharHaddad you are right! $\endgroup$ – Esmailian Mar 16 '19 at 18:09
  • 1
    $\begingroup$ @Esmailian thanks for updating your answer. Your contribution to data science group is highly appreciated, but making sure that we are providing the right answers is extremely important. Thank you one more time for updating your answer $\endgroup$ – Bashar Haddad Mar 17 '19 at 10:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.