1
$\begingroup$

According to this article, "Systems used for intent classification contain the following two components: Word embedding, and a classifier." This article also evaluated BERT+SVM and Word2Vec+SVM.

I'm trying to do the opposite, comparing two different classifiers (RNN and SVM) using BERT's word embedding.

Most Python codes that I found use BERT for the whole intent classification problem which made me confused. Example

I only want to map the words into vectors with BERT and feed the result into a classifier (SVM/RNN). Does BERT support word embedding and text classification at the same time? Does someone has an explanation? Is what I'm trying to test feasible with Python?

I have a dataframe that has two columns: intent and questions. It's a small dataset.

Thank you!

$\endgroup$
  • $\begingroup$ You don't need to use an RNN/SVM once you're keyed into a BERT architecture. Your BERT model will generate embeddings and can be fine-tuned (ala ULMfit last layer) to perform a specific task. You could potentally just use the emebddings and then perform the task with another model but the performance would likely not be better. $\endgroup$ – Learning stats by example Aug 25 at 16:21
2
$\begingroup$

BERT is a transformer based model. The pre-trained BERT can be used for two purposes:

  1. Fine-tuning: This involves, fine-tuning with the new data to a specific task such as classification or question-answering, etc. Here, the BERT itself acts like a classifier.
  2. Extracting embeddings: Here, you can extract the pretrained embeddings. The difference between Word2Vec (or other word embeddings) and BERT is that BERT provides contextual embeddings, meaning, the embeddings of each word depends on its neighbouring words. However, since it's contextual embeddings, we can make an assumption that the fist token which is '[CLS]' captures the context can be treated as sentence embeddings as be used as input to 'SVM' or other classifer. But, in the case of RNN, you may want to take the pretrained embeddings of each token to form a sequence.

So, how you want to use BERT still remains a choice. But if you can fine-tune the BERT model, it would generally yield higher performance. But you'll have to validate it based on the experiments.

| improve this answer | |
$\endgroup$
0
$\begingroup$

You can surely use BERT's word embedding with SVM/RNN. Essentially, this is what transfer learning aims at. BERT's MLM approach makes it one of the most reliable choice for word embedding.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.