1
$\begingroup$

Could somebody point me to a paper or code that is about classifying texts into potentially thousands of categories (topics)? I do have data based on Wikipedia and the number of categories is really big (thousands), so looking for some solutions that should work.

$\endgroup$
  • 1
    $\begingroup$ what did you try so far? $\endgroup$ – Peter Apr 2 at 11:18
2
$\begingroup$

In general this doesn't work well, since it's almost unavoidable that the classifier won't be able to distinguish all the categories from each other. I'd suggest tying to reduce the number of categories (for instance discard the least common ones).

In any case I'm not aware of any specific model to deal with a high number of classes, it's regular text classification. I'd suggest to start with a robust method such as decision trees, but there are many options.

| improve this answer | |
$\endgroup$
1
$\begingroup$

Currently BERT is the preferred method for many NLP tasks. GRUs are also used. https://medium.com/huggingface/multi-label-text-classification-using-bert-the-mighty-transformer-69714fa3fb3d

There are some other methods that have been discussed which I haven't tried. https://paperswithcode.com/task/text-classification

About the issue of a classifier being unable to determine a certain class, might be true to some extent. This often happens when you have a class that has significantly fewer instances than the rest of classes.

Text classification with thousands of output classes in Keras

It all depends on the quality of data you are working with and the amount of processing power available.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.