How do you apply SMOTE on text classification?

Question

Synthetic Minority Oversampling Technique (SMOTE) is an oversampling technique used in an imbalanced dataset problem. So far I have an idea how to apply it on generic, structured data. But is it possible to apply it on text classification problem? Which part of the data do you need to oversample? There is already another question about it, but it doesn't have an answer. Where can I possibly learn to get started with this?

Answer 1

Actually NLP is one of the most common areas in which resampling of data is needed as there are many text classification tasks dealing with imbalanced problem (think of spam filtering, insulting comment detection, article classification, etc.). But SMOTE seem to be problematic here for some reasons:

• SMOTE works in feature space. It means that the output of SMOTE is not a synthetic data which is a real representative of a text inside its feature space.
• On one side SMOTE works with KNN and on the other hand, feature spaces for NLP problem are dramatically huge. KNN will easily fail in those huge dimensions.

So I can propose you two approaches:

1. Do not care about the real text representation of new synthetic samples which I assume should be fine. You need to balance the distribution for your classifier not for a reader of text data. So apply SMOTE as traditional (however I usually use the solution 2 bellow so I do not gaurantee the result!) with some Dimensionality Reduction step.

1) Lets assume you want to make your data samples from minor class double using 3-NN. Ignore the major class(es) and keep only minor class samples.

2) For each sample point in feature space choose 5 nearest neighbors. Then choose 3 of them randomly (isn't it unnecessary complicated? if I didn't want to explain the original algorithm I would say just choose 3 neighbors!)

3) For each dimension calculate the distance between sample and neighbors and multiply it in a random number between 0-1 and add it to the original value of sample in that dimension. (this complicated paragraph simply means for each dimension choose a random value between the original sample and that neighbor!)

2. But I usually do another oversampling which is on the text (so more intuitive) and is kind of SMOTE.

1) Ignore the major class. Get a length distribution of all documents in minor class so that we generate new samples according the the true document length (number of words/phrases). We assume we want to make the size of class triple (so producing $k=2$ synthetic documents per original document)

2) Generate a sequence of $n$ random integers according to that distribution. It's used for determining the length of new synthetic document.

3) For each document: Choose one integer from random length sequence and $m$ random document whose length is close to the integer. Put tokens of all $m$ documents in a set and randomly choose $n$ tokens $k$ times. these are your $k$ new documents.

Answer 2

If you want to add more text/sentences traning data, you can use pre-trained word embeddings. Pretrained models like provides word vector representation of each and every dictionary word. It also provides 'most_smiliar' words.

You can simply use sample sentence and generate new samples by doing permutations of each and every word with its most TOP_N similar words.

e.g. if your sentence has 3 words, and if you choose 3 most similar for each words, you can upsample by 27 times.

Answer 3

I am having the same problem. I have multi-classification problem with imbalanced dataset over 10k for majority and 200 for minority.

Tired SMOTE but later have problem like Kasra Manshaei mention.

Text/Data augmentation is a method that I am trying. You basically take a sample(text) from dataset and replace some words with its synonym in the text to create a synthetic text. This way you have new synthetic text and after vectorisation the data still represent the original text.

Reference

nlpaug is a library that can handle text augmentation.