0
$\begingroup$

I am wondering the calculation of perplexity of a language model which is based on character level LSTM model. I got the code from kaggle and edited a bit for my problem but not the training way. I have added some other stuff to graph and save logs. However, as I am working on a language model, I want to use perplexity measuare to compare different results. In tensorflow, I have done it via this answer and it was easy. I have looked for a way doing it in PyTorch and literally no related result on Google. I need some help, and it is really appreciated.

Here is the related code, I believe:

criterion = nn.CrossEntropyLoss()

# create training and validation data
val_idx = int(len(data)*(1-val_frac))
data, val_data = data[:val_idx], data[val_idx:]

if(train_on_gpu):
    net.cuda()

counter = 0
n_chars = len(net.chars)
for e in range(epochs):
    # initialize hidden state
    h = net.init_hidden(batch_size)

    for x, y in get_batches(data, batch_size, seq_length):
        counter += 1

        # One-hot encode our data and make them Torch tensors
        x = one_hot_encode(x, n_chars)
        inputs, targets = torch.from_numpy(x), torch.from_numpy(y)

        if(train_on_gpu):
            inputs, targets = inputs.cuda(), targets.cuda()

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients
        net.zero_grad()

        # get the output from the model
        output, h = net(inputs, h)

        # calculate the loss and perform backprop
        loss = criterion(output, targets.view(batch_size*seq_length))
$\endgroup$
1
$\begingroup$

I was surfing around at PyTorch's website and found a calculation of perplexity. You can examine how they calculated it as ppl as follows:

criterion = nn.CrossEntropyLoss()
total_loss = 0.
...
for batch, i in enumerate(range(0, train_data.size(0) - 1, bptt)):
    ...
    loss = criterion(output.view(-1, ntokens), targets)
    loss.backward()
    total_loss += loss.item()
    log_interval = 200
    if batch % log_interval == 0 and batch > 0:
        cur_loss = total_loss / log_interval
        ...
        print('ppl {:8.2f}'.format(math.exp(cur_loss)))
        ...
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.