0
$\begingroup$

I am aware (continuously learning) of the advantages of Transformers over LSTMs.

At the same time, I was wondering from the viewpoint of size of the data needed, contrast of those two techniques, supposing I want to train for a downstream task, (classification or NER for instance), in which case would I need more data to achieve a specific result (although I am fully aware we never know in advance for a task how much data we need).

Presuming a result of N% (supposing that threshold is achievable for both LSTM and BERT), which architecture (LSTM or BERT) would require a bigger dataset (regardless of the size, I am aware dataset size is task-dependent and subject to change) to reach that point.

Does BERT need a bigger dataset to achieve "good results" (an empirical observation would help me) or a, say, bidirectional LSTM?

$\endgroup$
8
  • 1
    $\begingroup$ It does not work that way. You can't know a prior how much data you need to achieve a specific accuracy. $\endgroup$
    – noe
    Aug 5 at 8:45
  • $\begingroup$ I know, I was just trying to get "a feeling"/empirically on the amount of data needed between LSTMs and BERT, being aware that we do not know how much data we need to achieve a specific accuracy. $\endgroup$ Aug 5 at 8:46
  • $\begingroup$ I am only curious in the amount of dataset (regardless of the amount) in principle, empirically observed, by someone who has trained both LSTMs and Transformers, that is all (the contrast between those two, not that I need 4000 balanced dataset for sentiment classification), which of those 2 "needs more data" in order to achieve good results. $\endgroup$ Aug 5 at 8:49
  • $\begingroup$ I do not think/know how to express more clearly what I am trying to say. $\endgroup$ Aug 5 at 8:49
  • $\begingroup$ You should also know that comparing BERT and LSTMs is not a fair comparison, as with BERT you are doing transfer learning, so it would profit from the pre-trained data, not only from your training data. $\endgroup$
    – noe
    Aug 5 at 8:53
1
$\begingroup$

You can't know a prior how much data you need to achieve a specific accuracy. However, if you just want to know how much people have achieved on other datasets, you can check BERT's original paper:

enter image description here

As you can see, there are some datasets of 3.5k and 2.5 examples.

You should also know that comparing BERT and LSTMs is not a fair comparison, as with BERT you are doing transfer learning, so it would profit from the pre-trained data, not only from your training data.

$\endgroup$
1
  • $\begingroup$ That 3.5k and 2.5 examples was of importance to me. Thank you for the table. $\endgroup$ Aug 5 at 12:00
1
$\begingroup$

If you use pretrained Transformer, then you might need very very small dataset. ( I have achieved good accuracy with as low as 100 training sample with a positive/negative sentiment classification). But if you try to train a transformer from scratch , it will require a huge dataset.

Similarly, if you use pretrained Word2Vec/Glove to embed texts and use a LSTM network to train, then you can get good accuracy with very small dataset (again as low as 100 sample for binary classification). But training a Word2Vec from scratch will require a decent amount of training sample.

If we compare training from scratch for both cases, from my experience, transformer will require much larger dataset than LSTM.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for the comment. Indeed I expect that transformers would also need a much bigger dataset than LSTM. I was thinking as a comparison viewpoint between "transfer learning" on a downstream task, starting from the assumption that we have pre-trained an LSTM on huge dataset as well as a BERT on a huge dataset. $\endgroup$ Aug 5 at 9:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.