1
$\begingroup$

The question is based on Andrej Karpathy lecture on training a toy GPT model on Tiny Shakespeare dataset (youtube link). In this model, tokens are single characters with a dictionary of around 60 elements. He creates training and validation datasets as follows

def get_batch(...):
    ...
    ix = torch.randint(len(data) - block_size, (batch_size,))
    x = torch.stack([data[i:i+block_size] for i in ix])
    y = torch.stack([data[i+1:i+block_size+1] for i in ix])
    ...
    return x, y

Why does he make the target output y a sequence of bytes data[i+1:i+block_size+1] instead of just a single byte data[i+block_size+1]? We are trying to predict the next single byte (token) after all.

It looks to me like gpt is trained to predict N (N is the block_size) characters of Y from N characters of X, but the first (N-1) characters in Y are just a copy of (N-1) characters in X. Surely the NN can be trained to do that, but isn't it a waste of weights and GPU cycles on essentially just copying (N-1) characters?

I ran the script in debugger to confirm that's what indeed happens in training. In the code, after the first breakpoint

class GPTLanguageModel(nn.Module):

    ...
    def forward(self, idx, targets=None):
>>>     B, T = idx.shape

idx is something like tensor([[ 1, 16, 24, 40, 26, 32, 43, 41],...]), while target is tensor([[16, 24, 40, 26, 32, 43, 41, 1],...])

$\endgroup$

1 Answer 1

1
$\begingroup$

Language models predict the next token. At training time, we have an input sequence and we want to train the model to output the next token for each possible sequence prefix at the same time, that is, we train the model to generate token 1 based on nothing, token 2 based on token 1, token 3 based on tokens 1 and 2, etc, everything at once. For that, the target output is the same as the input but shifted one position.

Note that, due to the self-attention mask of the Transformer model, the prediction at each position only depends on the previous tokens.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.