# Why does an attention layer in a transformer learn context?

I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers.

What I'm confused on is why the output of an attention layer is a context vector. That is to say: what is it about the way that a transformer is trained causes the attention layers to learn context? What I would expect to see in the paper is a justification along the lines of "when you train a transformer using attention on sequence-to-sequence tasks, the attention layers learn context and here's why...". I believe it because I've seen the heatmaps that show that attention between related words, but I want to understand why that is necessarily the result of training a transformer.

Why couldn't it be the case that the attention layers learn some other features that happen to also be beneficial in sequence to sequence tasks? How do we know that they learn context, other than that's what we observe?

Again, I get the math and I know there are several posts about it. What I want to know is what about the math or the training process implies that the attention layers learn context.

• what is your definition of the context here? In the most general sense context is just any kind of information that can help you to interpret something in a way it was to supposed to be interpreted. why that is necessarily the result of training a transformer - to that point, what else can it be result of?
– Deil
May 25, 2021 at 13:54
• Awesome. Your second sentence: "In the most general sense context is just any kind of information that can help you to interpret something in a way it was to supposed to be interpreted" really gets at the core of the thing I was stuck on. Jun 8, 2021 at 16:31