6
$\begingroup$

I have a small dataset of 977 rows with a class proportion of 77:23.

For the sake of metrics improvement, I have kept my minority class ('default') as class 1 (and 'not default' as class 0).

My input variables are categorical in nature. So, the below is what I tried. Let's assume we don't have age and salary info

a) Apply encoding like rare_encoding and ordinal_encoding to my dataset

b) Split into train and test split (with stratify = y)

c) Apply SMOTE to resample the training data only.

However, my question is on how does SMOTE work/resample when there is only categorical variable like below

gender  degree    occupation    Country     status
 MALE    BE        ENGGINER      USA        default
 MALE    ME        RESEARCHER    UK         default
 FEMALE  BSc       Admin staff   NZ         default  
 FEMALE  MS        Scientist     sweden     default 

Now if my objective is to oversample minority sample using SMOTE, How will the above sample look like? Will they just randomly populate/shuffle gender, degree, occupation and country on different permutation and combinations?

Is there any simple explanation or tutorial that you can share for someone who likes to applying this technique?

My objective is to understand how does SMOTE work for categorical variables only dataset?

$\endgroup$
3
  • 1
    $\begingroup$ While this does not directly address your question, statisticians do not see class imbalance as such a problem, and there is no need to use artificial balancing to solve a non-problem. It might be helpful if you say why you find the imbalance problematic. stats.stackexchange.com/questions/357466 fharrell.com/post/class-damage fharrell.com/post/classification stats.stackexchange.com/a/359936/247274 stats.stackexchange.com/questions/464636 twitter.com/f2harrell/status/1062424969366462473?lang=en $\endgroup$
    – Dave
    Feb 20, 2022 at 11:35
  • $\begingroup$ @Dave - yes, agree. I did read the posts that you linked... but with imbalance, we see poor performance for minority class. So, instead of optimizing the threshold, i would like to see whether upsampling can work...because this is one of the widely used solutions to solve this problem by the ML community (mainly for prediction purpose). But yes, understand that statisticans dont agree with this $\endgroup$
    – The Great
    Feb 20, 2022 at 11:40
  • $\begingroup$ @Dave - I would also welcome any suggestions from you on why sampling shouldn't be done. Because I feel there is 2 group of people. One who focus on statistics etc and other group focuses on predictions. For example, these synthetic data generation approavhes are developed by researchers with some statistics background as well (from their papers). Since, all this is like experiments, am just trying dofferent aplroaches mainly to see whether it improves performance. If yes, its useful. If not, not useful for my case. $\endgroup$
    – The Great
    Feb 20, 2022 at 12:17

1 Answer 1

3
$\begingroup$

SMOTE itself wouldn't work well for a categorical only feature-set for a few reasons:

  1. It works by interpolating between different points. Depending on how the data is encoded, you might end up with some undefined class (when using one-hot encoding, you might end up with a point that is half of one class and half of another class), or you might end up with a correct class but it doesn't make any sense from an interpolation point of view (for example, if you encode for example the country on a numerical scale like 1 -> US, 2 -> UK, 3 -> NZ, but it doesn't make much sense to interpolate between US and NZ and end up in UK).
  2. SMOTE uses k-means to select points to interpolate between. If you encode your categorical features using one-hot-encoding, you typically end up with a lot of sparse dimensions (dimensions that most points take only the value 0 in). k-means typically won't perform very well in such a space, and points that are nearby in this space might not look a lot like each other.

What you can do is use a modification of the SMOTE algorithm, called SMOTE-N (see https://imbalanced-learn.org/dev/over_sampling.html#smote-variants), which works when all features are categorical. This modifies the SMOTE algorithm to

  1. Use a different interpolation method: selects the most common class of the nearest neighbors
  2. Use a different distance metric (Value Difference Metric) instead of Euclidean distance in the encoded space.

In that link this method is attributed to the original SMOTE paper (https://www3.nd.edu/~dial/publications/chawla2002smote.pdf) where it's found in Section 6.2. There is also SMOTE-NC which is a combination of SMOTE and SMOTE-N for data which has both numerical and categorical features.

For your example, let's say for some reason 3 of the points given: MALE ME RESEARCHER UK default FEMALE BSc Admin staff NZ default
FEMALE MS Scientist sweden default

are considere nearby each other and are used for interpolation. Then a possible added point by SMOTE-N would be:

  • FEMALE (because that's the majority class)
  • MS (all 3 classses have equal frequency, so a class is randomly picked)
  • RESEARCHER (idem to above)
  • NZ (idem to above)
  • default (majority class)
$\endgroup$
2
  • $\begingroup$ thanks, upvoted for the help. So, for SMOTE or its variants (SMOTE-NC, SMOTE-N) to work, we definitely need a mix of numeric and categorical variables. There is no way we can upsample our data with only categorical variables? $\endgroup$
    – The Great
    Feb 20, 2022 at 14:27
  • $\begingroup$ SMOTE-N should work with only categorical variables, I'll clarify in the answer. $\endgroup$ Feb 20, 2022 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.