My understanding is that transformer decoders and transformer encoder-decoder models typically operate in the way that the GPT-2 does, i.e., representations in the generated sequence are computed once and then reused for future steps. But you are correct that this is not the only way things can be done. One could recompute the representations for all tokens in the partially-generated sequence using full self-attention over the tokens in the sequence generated so far (there's no mathematical hindrance to doing this -- it's akin to running a typical transformer encoder over the sequence of words in the partially-generated sequence).
But this additional computation is not commonly done as far as I can tell from the literature. I think there are at least two reasons. First, as noted by others, it's cheaper computationally to use previously-computed representations from earlier time steps (though it leads to different results, and I have not seen an empirical comparison in any papers). Second, it matches how training is done. During training, a consequence of masking in self-attention is that the representation at output position i is computed using representations at output positions <= i. That means that during training, there is only a single representation computed for output position i for each layer. That matches what happens at inference time using the standard approach that we've been discussing and which is used in the GPT-2.
If we wanted to train a model in which the representation for an output position was computed based on all available output representations (always excluding the ones that have not yet been "generated" of course), then we would need to compute multiple representations for each output position during training, one for each possible uncovered partial right context. For example, if we train a language model on windows of size 512, we would have to compute (about) 512 representations for the first word, one corresponding to the loss for generating each subsequent word in the window. This would lead to a very large computation graph and slow down training. However, it may work better because it leads to richer output representations, so please try it and let us know. :)