1
$\begingroup$

How BERT model differentiate between words with different meanings e.g. #Transformers like name of the movie or name of a library by @huggingface?

$\endgroup$

1 Answer 1

1
$\begingroup$

BERT models work with sentences, not words: the self-attention in the transformer architecture is considering each token in respect to all other tokens in the sentence.

"I'm going to implement Transformers in Python" vs. "I'm going to watch Transformers on TV later tonight" - there should be enough surrounding context to distinguish them. I.e. the John Firth quote: "You shall know a word by the company it keeps."

Note that the output of the the embedding layer (which is usually fed in to the first transformer layer) is the same for identical tokens (*): at this point the model has no way to tell the difference. It is only the interaction with the other tokens, as it passes through each layer, that enable a distinction to emerge.

Being able to distinguish between them also relies on the model having seen enough training sentences where each sense of a word was used.

*: pedantically, position codes are added in, before feeding into layer 1.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.