5
$\begingroup$

I am completely new to NLP and I have been tasked with performing text classification on a dataset containing 193k records. The number of classes is 107.

The class with the highest number of records contains > 16k entries, whereas the less frequent one contains only 5. You can see the frequency distribution below. The class names have been redacted due to confidentiality requirements.

class frequency distribution

Each entry can contain up to 100 characters. The text is very terse and contains few words in English, with the remaining being codes, locations and names of people.

How would you tackle such a problem? Does it make any sense to do text augmentation, or should I implement some form of weighting at the model evaluation stage? If so, which text augmentation / weighing tools or procedures would you recommend?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$
  • With 200k instances, a class which has less than 10 instances represents less than 0.005% of the data. It's very unlikely that a model can learn to distinguish all these classes, especially the smallest ones.
  • Data augmentation is really not recommended with text. Text is very diverse so there's no way to generate a good representative sample from a few instances. This would only lead to either repeating almost the same instances (pointless) or generating made up data which doesn't really represent the class (biased dataset).

Realistically, it's unlikely that anything can be done with the very small classes. What I'd suggest is to start by training a model with only the top 5 or 10 classes, and then improve from there if it works reasonably well. Note that short texts are often difficult to classify, since they might contain too little information. As a simple rule of thumb, if a human expert looking at an instance cannot find which class it belongs to, then chances are that a ML model cannot either.

Improve this answer
$\endgroup$
1
  • $\begingroup$ nice 2nd point explaining the reason for not applying data augmentation $\endgroup$
    – German C M
    Sep 16, 2021 at 15:12
2
$\begingroup$

I faced a similar situation regarding the imbalance among all possible classes when building a topics classifier in my current company, in this case for a callcenter conversations dataset.

As @Erwan also suggests, I would try to:

  • focus first on a model for the most frequent classes (as analogy, in my case it was about modeling first only the main topics); in case you see this first approach is realiable enough or above a baseline model, you could try to:
  • go on a model with more classes, but still grouping the ones less representative in a label called "other"
  • as a following step, in my case there were some generic topics (the most frequent ones, in your case the most frequent labels) and a specific model was built per topic to detect related sub-topics (in case you can classify your labels as generic and secondary labels...).

Regarding the length of your texts, I suggest, for the modeling phase, to take into account this experimental rule of thumb by François Chollet:

enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.