Could somebody point me to a paper or code that is about classifying texts into potentially thousands of categories (topics)? I do have data based on Wikipedia and the number of categories is really big (thousands), so looking for some solutions that should work.
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ what did you try so far? $\endgroup$– PeterCommented Apr 2, 2020 at 11:18
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
In general this doesn't work well, since it's almost unavoidable that the classifier won't be able to distinguish all the categories from each other. I'd suggest tying to reduce the number of categories (for instance discard the least common ones).
In any case I'm not aware of any specific model to deal with a high number of classes, it's regular text classification. I'd suggest to start with a robust method such as decision trees, but there are many options.
$\begingroup$
$\endgroup$
Currently BERT is the preferred method for many NLP tasks. GRUs are also used. https://medium.com/huggingface/multi-label-text-classification-using-bert-the-mighty-transformer-69714fa3fb3d
There are some other methods that have been discussed which I haven't tried. https://paperswithcode.com/task/text-classification
About the issue of a classifier being unable to determine a certain class, might be true to some extent. This often happens when you have a class that has significantly fewer instances than the rest of classes.
Text classification with thousands of output classes in Keras
It all depends on the quality of data you are working with and the amount of processing power available.
