Bert and SVM classification

Question

I'm trying to understand the concepts in the title and how they fit into the task of binary classification. According to my understanding so far, you can encode text using various feature-extraction methods such a bag of words. You can then use something like liblinear to obtain a SVM LibLinear model that is able to classify your data. On the other hand, you can build a model by concatenating Bert with a Dense layer. You can then fine tune this model and again, you obtain a classifier. Where would you use either one of them and why?

Answer 1

BERT is a transformer-based model. The pre-trained BERT can be used for two purposes:

1. Fine-tuning
2. Extracting embedding

You don't need to use an SVM once you're keyed into a BERT architecture. Your BERT model will generate embeddings and can be fine-tuned (ala ULMfit last layer) to perform a specific task. You could potentially just use the embeddings and then perform the task with another model but the performance would likely not be better.

So, how you want to use BERT still remains a choice. But if you can fine-tune the BERT model, it would generally yield higher performance. But you'll have to validate it based on the experiments.