Transformers (BERT) vs LSTM on Sentiment Analysis/NER - dataset sizes comparison

Question

I am aware (continuously learning) of the advantages of Transformers over LSTMs.

At the same time, I was wondering from the viewpoint of size of the data needed, contrast of those two techniques, supposing I want to train for a downstream task, (classification or NER for instance), in which case would I need more data to achieve a specific result (although I am fully aware we never know in advance for a task how much data we need).

Presuming a result of N% (supposing that threshold is achievable for both LSTM and BERT), which architecture (LSTM or BERT) would require a bigger dataset (regardless of the size, I am aware dataset size is task-dependent and subject to change) to reach that point.

Does BERT need a bigger dataset to achieve "good results" (an empirical observation would help me) or a, say, bidirectional LSTM?

Answer 1

You can't know a prior how much data you need to achieve a specific accuracy. However, if you just want to know how much people have achieved on other datasets, you can check BERT's original paper:

As you can see, there are some datasets of 3.5k and 2.5 examples.

You should also know that comparing BERT and LSTMs is not a fair comparison, as with BERT you are doing transfer learning, so it would profit from the pre-trained data, not only from your training data.

Answer 2

If you use pretrained Transformer, then you might need very very small dataset. ( I have achieved good accuracy with as low as 100 training sample with a positive/negative sentiment classification). But if you try to train a transformer from scratch , it will require a huge dataset.

Similarly, if you use pretrained Word2Vec/Glove to embed texts and use a LSTM network to train, then you can get good accuracy with very small dataset (again as low as 100 sample for binary classification). But training a Word2Vec from scratch will require a decent amount of training sample.

If we compare training from scratch for both cases, from my experience, transformer will require much larger dataset than LSTM.