2
$\begingroup$

I'm trying to solve a multilabel classification (dataset is tweet text) using a combination of BERT and CNN. As a benchmark, I'd compare it to other word embeddings, one of which is Word2vec. After numerous tries, it seems that Word2vec-CNN keeps outperforming BERT-CNN by a slight bit, here's a result from my last try:

Word2vec-CNN
precision (macro): 0.89  
recall (macro): 0.87  
f1-score (macro): 0.88
accuracy (test set): 0.81
hamming loss: 0.062

BERT-CNN
precision (macro): 0.86  
recall (macro): 0.88  
f1-score (macro): 0.87
accuracy (test set): 0.74
hamming loss: 0.073

Question is:

  1. Could it be possible that Word2vec (or any static word embeddings) outperforms BERT (or any contextual word embeddings)? If so, what is the rationale? If there's any research paper on this it would be really helpful.
  2. If not, what could possibly be the cause?

FWIW: Model is trained using TensorFlow-Keras (I kind of suspect this is SOMEHOW caused by how TF-Keras calculates its metrics but I still haven't figured out why and, if any, a solution), and both embeddings are pretrained (BERT model was trained on a bigger corpus, around 200:1).

$\endgroup$
4
  • 3
    $\begingroup$ If a non-contextual embedding approach outperforms a contextual embedding approach in certain task, a possible cause is that context is not that important for such a task. $\endgroup$ – noe Apr 18 at 17:10
  • $\begingroup$ @noe Could it be said that the dataset is not giving enough context for it to be meaningful? I did notice performance went down when I applied additional preprocessing methods. $\endgroup$ – Neruda Apr 18 at 23:54
  • 1
    $\begingroup$ I understand that the cause could be many different things (e.g. a software bug), but I don't see how we could possibly derive such a conclusion (i.e. "the dataset is not giving enough context for it to be meaningful") from the information you described. $\endgroup$ – noe Apr 19 at 6:39
  • $\begingroup$ Did you finetune BERT as well? If yes, it can be also overfitting. $\endgroup$ – Jindřich Apr 19 at 7:59
1
$\begingroup$

Yes, this could be possible if your dev/test data comes from the same domain as the training data, in which case word2vec will encounter fewer OOV tokens that mess up the loss.

This could also mean that the benefits of BERT - subword tokenization to handle OOV characters in generalized domains - are lost. If your vocabulary size is small, your word2vec model needs to capture relationships between fewer tokens and can model those relationships better than a subword model which loses the relationships between fixed tokens in your data and instead tries to generalize relationships across >30K subword tokens (in the bert-based-uncased model), which could lead to noise.

$\endgroup$
1
$\begingroup$

Glad you found where it went wrong! However, it is really possible for something like that to happen. There is no such thing as "best algorithm", so the performance of a method partly depends on what your dataset looks like. Or sometimes your feature engineering method just allows the data to cheat on you, say, you mistakenly leaked some data, or neglected the imbalanced nature of the dataset.

$\endgroup$
0
$\begingroup$

Thanks to everyone who gave answer and comments. It was indeed caused by my data.

  1. Prior to this I had the same preprocessing pipeline for both models, which would be your "usual" NLP preprocessing steps (non-alphanumerical removal, lowercasing, stemming, and stop word removal). I had a hunch that both stemming and stop word removal would cause the text to lose some context and hence the benefits of BERT wouldn't be prevalent. Experimenting with another preprocess pipeline that DOES NOT do stemming and stop word removal actually proved to benefit the BERT model, as shown from the metrics below. Further reading: this article and this paper.

  2. My data DID come from the same domain as the training data for the pretrained Word2vec (i.e. tweets). On the other hand, the pretrained BERT was trained on a combination of tweets and other kinds of texts. In addition to (1), I conclude it would be much easier for Word2vec to model the texts. Further reading: this paper.

Metrics

Word2vec-CNN
precision (macro): 0.87
recall (macro): 0.87  
f1-score (macro): 0.87
accuracy (test set): 0.83
hamming loss: 0.063

BERT-CNN
precision (macro): 0.91  
recall (macro): 0.90  
f1-score (macro): 0.90
accuracy (test set): 0.82
hamming loss: 0.051
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.