1
$\begingroup$

I've studied about various text representation techniques like : Bag of Words, N-gram data modelling, Tf-idf, word embedding etc.

I would like to know which among all the techniques are most efficient when it comes to data modelling or representation for a supervised text classification across a large number of categories.

I might have around 40 categories and then around a same number of sub-categories upto 4 levels.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

There's no simple answer to this question. As far as I know in general the choice depends mostly on the type of classification:

  • Bag of Words (usually with tf-idf weights) is a simple but quite efficient representation for classification based on the text topic or similar, assuming the classes are reasonably distinct from each other.
  • Word embeddings are a more advanced option for semantic-based classification. They can handle more subtle semantic relations but require being trained on a large training corpus. Using pre-defined embeddings can be a solution but then there's the risk that the original training data isn't perfectly suitable for the dataset.
  • N-grams models can be used in many different ways but are often chosen when the classification involves syntax and/or writing style. Note that the higher the value $n$, the larger the training corpus needs to be, this can also be taken into account in the choice.

I might have around 40 categories and then around a same number of sub-categories upto 4 levels.

It depends on the data but 40 classes is already a very challenging classification task. For the sake of simplicity let's assume a uniform distribution over classes: a random baseline accuracy would be 1/40 = 2.5%. Of course it depends on the data and a good classifier will do better than that, but don't expect too much...

Now 4 levels of 40 sub-categories means 40^4 = 2.5 millions classes! Even assuming you have enough data (say around 10 instances by class in average, that is 25 millions instances!), it's very unlikely that a classifier will be able to predict anything useful from such a huge amount of classes.

Improve this answer
$\endgroup$
5
  • $\begingroup$ What if I start with around 130 classes, what approach should I use to move forward? Actually, I'm new to NLP. Although I'm studying theory and well understanding it but facing trouble when it comes to deciding an approach. $\endgroup$
    – ashmit-001
    Jan 28, 2020 at 6:26
  • 1
    $\begingroup$ @mercury-01 in my experience it's better to start simple, here I would say with tf-idf vectors and something like decision trees for instance (decision trees have the advantage that you can see how the model makes a decision). Then you can analyze what happens specifically with the data and decide which direction to go next. $\endgroup$
    – Erwan
    Jan 28, 2020 at 12:10
  • $\begingroup$ Thanks @Erwan, I'll start with this algorithm and share the results if I found something benefitial. $\endgroup$
    – ashmit-001
    Jan 29, 2020 at 5:17
  • $\begingroup$ ngram to my understanding is mostly for next word prediction. How can you use ngram to represent texts in varying lengths? $\endgroup$
    – Student
    Apr 3, 2022 at 10:03
  • $\begingroup$ @hing it's the traditional bag of words representation. That's what was used for language models for instance. $\endgroup$
    – Erwan
    Apr 3, 2022 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.