I'm trying to solve a multilabel classification (dataset is tweet text) using a combination of BERT and CNN. As a benchmark, I'd compare it to other word embeddings, one of which is Word2vec. After numerous tries, it seems that Word2vec-CNN keeps outperforming BERT-CNN by a slight bit, here's a result from my last try:
Word2vec-CNN precision (macro): 0.89 recall (macro): 0.87 f1-score (macro): 0.88 accuracy (test set): 0.81 hamming loss: 0.062 BERT-CNN precision (macro): 0.86 recall (macro): 0.88 f1-score (macro): 0.87 accuracy (test set): 0.74 hamming loss: 0.073
- Could it be possible that Word2vec (or any static word embeddings) outperforms BERT (or any contextual word embeddings)? If so, what is the rationale? If there's any research paper on this it would be really helpful.
- If not, what could possibly be the cause?
FWIW: Model is trained using TensorFlow-Keras (I kind of suspect this is SOMEHOW caused by how TF-Keras calculates its metrics but I still haven't figured out why and, if any, a solution), and both embeddings are pretrained (BERT model was trained on a bigger corpus, around 200:1).