Sentiment analysis BERT vs Model from scratch

Question

I am working on building a sentiment analyzer, the data I would like to analyze is social media data from twitter, once I have created a the model I want to integrate it into a simply webpage.

I have tried two options:

1. Create my own model from scratch, meaning train a word2vec model to perform word embedding, convert my labelled dataset into vectors and train them using Logistic regression, Random forest or SVM.

2. Fine tune a BERT model using my dataset.

option 1.. Using word2vec and SVM I was able to get the following results:

          precision    recall  f1-score   support

0              0.74      0.67      0.70      1310
1              0.77      0.82      0.79      1716

accuracy                           0.76      3026
macro avg       0.75      0.75      0.75      3026
weighted avg    0.75      0.76      0.75      3026

option 2.. I fined tuned BERT using the following code link and was able to achieve the following results after 100 epochs:

          precision    recall  f1-score   support

       0       0.68      0.65      0.66       983
       1       0.74      0.77      0.75      1287

accuracy                           0.71      2270
macro avg      0.71      0.71      0.71      2270
weighted avg   0.71      0.71      0.71      2270

I used the same dataset for both option 1 and 2, BERT used a smaller subset for validation

What I would like to know:

Is there any advantages in going with option 1.? Does BERT have any disadvantages when it comes to data from social media (data is rather unclean and a lot of slang).

Answer 1

In general, BERT is a much stronger model. Word embeddings only represent isolated words, whereas BERT considers the sentence context and how the words interact. With user-generated data, word embeddings might have plenty of OOV. In contrast, BERT uses subword tokenization that might partially compensate for that, although it is not ideal. It might be better to search for BERT-like models pre-trained specifically on social network data (e.g., TwHIN-BERT). The only disadvantage is higher computational complexity than classical machine learning over word embeddings.

As with any large model, it is prone to overfitting and catastrophic forgetting when not fine-tuned carefully. This might be the reason why you get slightly worse results with BERT. You can try smaller learning, training for fewer epochs, or freezing some layers.