Seems that in Vanilla transformers at least (a la AIAYN), during inference time, the hidden states are generated for all tokens in the input sequence, but only the last one is used to predict the next token.
My question - why are the other hidden states produced at all during inference? Wouldn't it be more efficient if only the last hidden state was produced?
For example, here : https://nlp.seas.harvard.edu/annotated-transformer/#greedy-decoding
out = model.decode( memory, src_mask, ys, subsequent_mask(ys.size(1)).type_as(src.data) ) prob = model.generator(out[:, -1]) _, next_word = torch.max(prob, dim=1) next_word = next_word.data