BERT is a Transformer encoder, while GPT is a Transformer decoder:
You are right in that, given that GPT is decoder-only, there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the masking in the multi-head attention block.
There is, however, an extra difference in how BERT and GPT are trained:
BERT is a Transformer encoder, which means that, for each position in the input, the output at the same position is the same token (or the
[MASK] token for masked tokens), that is the inputs and output positions of each token are the same.
GPT is a Transformer decoder, which means that it is meant for autoregressive inference. This means that the tokens in the input are shifted one position to the right with respect to the output, that is, if the output is [
</s>], the input is [
Apart from that, at inference time BERT generates all its output at once, while GPT is autoregressive, so you need to iteratively generate one token at a time.