Can we use BERT for only word embedding and then use SVM/RNN to do intent classification?

Question

According to this article, "Systems used for intent classification contain the following two components: Word embedding, and a classifier." This article also evaluated BERT+SVM and Word2Vec+SVM.

I'm trying to do the opposite, comparing two different classifiers (RNN and SVM) using BERT's word embedding.

Most Python codes that I found use BERT for the whole intent classification problem which made me confused. Example

I only want to map the words into vectors with BERT and feed the result into a classifier (SVM/RNN). Does BERT support word embedding and text classification at the same time? Does someone has an explanation? Is what I'm trying to test feasible with Python?

I have a dataframe that has two columns: intent and questions. It's a small dataset.

Thank you!

Answer 1

BERT is a transformer based model. The pre-trained BERT can be used for two purposes:

1. Fine-tuning: This involves, fine-tuning with the new data to a specific task such as classification or question-answering, etc. Here, the BERT itself acts like a classifier.
2. Extracting embeddings: Here, you can extract the pretrained embeddings. The difference between Word2Vec (or other word embeddings) and BERT is that BERT provides contextual embeddings, meaning, the embeddings of each word depends on its neighbouring words. However, since it's contextual embeddings, we can make an assumption that the fist token which is '[CLS]' captures the context can be treated as sentence embeddings as be used as input to 'SVM' or other classifer. But, in the case of RNN, you may want to take the pretrained embeddings of each token to form a sequence.

So, how you want to use BERT still remains a choice. But if you can fine-tune the BERT model, it would generally yield higher performance. But you'll have to validate it based on the experiments.

Answer 2

You can surely use BERT's word embedding with SVM/RNN. Essentially, this is what transfer learning aims at. BERT's MLM approach makes it one of the most reliable choice for word embedding.