# How to interepret BERT Attention

Can we tell BERT extracts local features? For example consider the sentence "This is my first sentence. This is my second sentence". Now How Bert extracts the features. attention is computed for each sentence or as whole?

BERT's self-attention will be computed for each pair of tokens. If the input sentence has $$N$$ tokens, then the attention weights will be computed over the $$N^2$$ pairs of tokens.

The attention, nevertheless, will be computed in each one of the attention heads of each of the layers of BERT.

If you want to understand the self-attention patterns that are normally found in BERT, you can check this article, where you will find analyses like this one:

• From given image, I can conclude that bert extracts global features. Is this conclusion right? Aug 2, 2021 at 15:05
• If with "global" you mean that the representation of each token depends also on the other tokens, the answer is yes.
– noe
Aug 2, 2021 at 15:38
• Does bert captures both local and global semantics Sep 4, 2021 at 8:01
• Yes, BERT captures both local and global information.
– noe
Sep 4, 2021 at 8:25
• but each token representation depends on all other tokens which extracts global information Sep 4, 2021 at 8:46