1
$\begingroup$

I try to identify applications of vanilla transformer in nlp, as well as those in BERT. But I don't seem to find good summaries for either of them. Thus my questions are:

  1. what are the applications of transformer and bert respectively?
  2. in (1), why in some application vanilla transformer is used over BERT? (or vice versa?) What're the reasons?

TIA.

$\endgroup$

1 Answer 1

1
$\begingroup$

A normal transformer has two parts: encoder (non-autoregressive) and decoder (autoregressive). This allows it to generate text (i.e. sequences of tokens). Therefore the applications of the vanilla transformer are those receiving a piece of text as input and getting another piece of text as output. The main example is machine translation.

BERT is a transformer encoder. Its applications are those tasks where the input is a piece of text (or N pieces of text) and the output is either:

  • One single output (at the [CLS] token position). This includes any classification or regression task.
  • One output per some/each of the input tokens. This mainly comprises token tagging tasks, e.g. part of speech tagging, span tagging (e.g. for question answering).
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.