I've done a bit of research, with this being the best as far as objectively measuring quality, but wanted to ask from a theoretical perspective if BoW-based models (e.g. using TF-IDF) or word embeddings-based models (e.g. Word2Vec) would ever be a better choice than a language model (e.g. BERT) for a text classification problem?
The specific problem I'm working on is binary classification of short 2-8 word snippets such as "Air bubble in ampoule" into categories "requires response" or "does not require response", but I'm more interested in the general question above.