I've done a bit of research, with this being the best as far as objectively measuring quality, but wanted to ask from a theoretical perspective if BoW-based models (e.g. using TF-IDF) or word embeddings-based models (e.g. Word2Vec) would ever be a better choice than a language model (e.g. BERT) for a text classification problem?

The specific problem I'm working on is binary classification of short 2-8 word snippets such as "Air bubble in ampoule" into categories "requires response" or "does not require response", but I'm more interested in the general question above.


Unfortunately, there is little theoretical knowledge about what complex neural networks do. Transformers are known to be universal approximations, so in theory they can learn to do any function with the input sentence, unlike the other alternatives that you mention. Most of the time, the accuracy of the BERT-like model would be strictly better.

In practice, however, everything depends on the data you have. Neural language models have very many parameters, which makes them often prone to overfitting and hard to train. Some classification problems might also be so easy that a stronger model would not help. There is also the question of computational efficiency, the accuracy gain might not be worth the slow-down from using a more complex model. BoW models might also offer better interpretability.

To conclude, there might be many situations and many reasons why smaller and simpler models might be a better choice.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.