I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers.
What I'm confused on is why the output of an attention layer is a context vector. That is to say: what is it about the way that a transformer is trained causes the attention layers to learn context? What I would expect to see in the paper is a justification along the lines of "when you train a transformer using attention on sequence-to-sequence tasks, the attention layers learn context and here's why...". I believe it because I've seen the heatmaps that show that attention between related words, but I want to understand why that is necessarily the result of training a transformer.
Why couldn't it be the case that the attention layers learn some other features that happen to also be beneficial in sequence to sequence tasks? How do we know that they learn context, other than that's what we observe?
Again, I get the math and I know there are several posts about it. What I want to know is what about the math or the training process implies that the attention layers learn context.