2
$\begingroup$

When BERT is used for masked language modeling, it masks a token and then tries to predict it.

What are the candidate tokens BERT can choose from? Does it just predict an integer (like a regression problem) and then use that token? Or does it do a softmax over all possible word tokens? For the latter, isn't there just an enormous amount of possible tokens? I have a hard time imaging BERT treats it like a classification problem where # classes = # all possible word tokens.

From where does BERT get the token it predicts?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

There is a token vocabulary, that is, the set of all possible tokens that can be handled by BERT. You can find the vocabulary used by one of the variants of BERT (BERT-base-uncased) here.

You can see that it contains one token per line, with a total of 30522 tokens. The softmax is computed over them.

The token granularity in the BERT vocabulary is subwords. This means that each token does not represent a complete word, but just a piece of word. Before feeding text as input to BERT, it is needed to segment it into subwords according to the subword vocabulary mentioned before. Having a subword vocabulary instead of a word-level vocabulary is what makes it possible for BERT (and any other text generation subword model) to only need a "small" vocabulary to be able to represent any string (within the character set seen in the training data).

Improve this answer
$\endgroup$
2
  • $\begingroup$ I'm surprised that it's possible to do a softmax over so many possible outcomes with any level of accuracy. But I suppose they're able to accomplish it because the set of training data is so large? $\endgroup$
    – Nick Koprowicz
    Commented Nov 17, 2020 at 3:57
  • 1
    $\begingroup$ Yes, as long as you have enough data, 30k elements in the softmax work well. For larger vocabularies (e.g. 100k), people often use adaptive softmax. Smaller training datasets lead to data sparsity, and this affects both the input embeddings and output projection+softmax. $\endgroup$
    – noe
    Commented Nov 17, 2020 at 9:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.