Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
I'm trying to do the opposite, comparing two different classifiers (RNN and SVM) using BERT's word embedding.
Most Python codes that I found use BERT for the whole intent classification problem which made me confused. Example
I only want to map the words into vectors with BERT and feed the result into a classifier (SVM/RNN).
Does BERT support word embedding and text classification at the same time? Does someone has an explanation? Is what I'm trying to test feasible with Python?
I have a dataframe that has two columns: intent and questions. It's a small dataset.
BERT is a transformer based model. The pre-trained BERT can be used for two purposes:
Fine-tuning: This involves, fine-tuning with the new data to a specific task such as classification or question-answering, etc. Here, the BERT itself acts like a classifier.
Extracting embeddings: Here, you can extract the pretrained embeddings. The difference between Word2Vec (or other word embeddings) and BERT is that BERT provides contextual embeddings, meaning, the embeddings of each word depends on its neighbouring words. However, since it's contextual embeddings, we can make an assumption that the fist token which is '[CLS]' captures the context can be treated as sentence embeddings as be used as input to 'SVM' or other classifer. But, in the case of RNN, you may want to take the pretrained embeddings of each token to form a sequence.
So, how you want to use BERT still remains a choice. But if you can fine-tune the BERT model, it would generally yield higher performance. But you'll have to validate it based on the experiments.