Can we tell BERT extracts local features? For example consider the sentence "This is my first sentence. This is my second sentence". Now How Bert extracts the features. attention is computed for each sentence or as whole?


1 Answer 1


BERT's self-attention will be computed for each pair of tokens. If the input sentence has $N$ tokens, then the attention weights will be computed over the $N^2$ pairs of tokens.

The attention, nevertheless, will be computed in each one of the attention heads of each of the layers of BERT.

If you want to understand the self-attention patterns that are normally found in BERT, you can check this article, where you will find analyses like this one:

enter image description here

  • $\begingroup$ From given image, I can conclude that bert extracts global features. Is this conclusion right? $\endgroup$ Aug 2, 2021 at 15:05
  • $\begingroup$ If with "global" you mean that the representation of each token depends also on the other tokens, the answer is yes. $\endgroup$
    – noe
    Aug 2, 2021 at 15:38
  • $\begingroup$ Does bert captures both local and global semantics $\endgroup$ Sep 4, 2021 at 8:01
  • $\begingroup$ Yes, BERT captures both local and global information. $\endgroup$
    – noe
    Sep 4, 2021 at 8:25
  • $\begingroup$ but each token representation depends on all other tokens which extracts global information $\endgroup$ Sep 4, 2021 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.