I am wondering the calculation of perplexity of a language model which is based on character level LSTM model. I got the code from kaggle and edited a bit for my problem but not the training way. I have added some other stuff to graph and save logs. However, as I am working on a language model, I want to use perplexity measuare to compare different results. In tensorflow, I have done it via this answer and it was easy. I have looked for a way doing it in PyTorch and literally no related result on Google. I need some help, and it is really appreciated.

Here is the related code, I believe:

criterion = nn.CrossEntropyLoss()

# create training and validation data
val_idx = int(len(data)*(1-val_frac))
data, val_data = data[:val_idx], data[val_idx:]


counter = 0
n_chars = len(net.chars)
for e in range(epochs):
    # initialize hidden state
    h = net.init_hidden(batch_size)

    for x, y in get_batches(data, batch_size, seq_length):
        counter += 1

        # One-hot encode our data and make them Torch tensors
        x = one_hot_encode(x, n_chars)
        inputs, targets = torch.from_numpy(x), torch.from_numpy(y)

            inputs, targets = inputs.cuda(), targets.cuda()

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients

        # get the output from the model
        output, h = net(inputs, h)

        # calculate the loss and perform backprop
        loss = criterion(output, targets.view(batch_size*seq_length))

1 Answer 1


I was surfing around at PyTorch's website and found a calculation of perplexity. You can examine how they calculated it as ppl as follows:

criterion = nn.CrossEntropyLoss()
total_loss = 0.
for batch, i in enumerate(range(0, train_data.size(0) - 1, bptt)):
    loss = criterion(output.view(-1, ntokens), targets)
    total_loss += loss.item()
    log_interval = 200
    if batch % log_interval == 0 and batch > 0:
        cur_loss = total_loss / log_interval
        print('ppl {:8.2f}'.format(math.exp(cur_loss)))

As @SpiderRico reminded, I got it from this link

  • 1
    $\begingroup$ Including source of this code for sake of the completeness. github.com/pytorch/examples/tree/master/word_language_model (I assume this is where the OP got the code). $\endgroup$
    – SpiderRico
    Mar 9, 2020 at 5:25
  • $\begingroup$ @SpiderRico Yes sir, you are correct. I have forgotten to add the link. Thanks for reminding. $\endgroup$
    – Faruk
    Mar 9, 2020 at 10:37
  • $\begingroup$ why it is divided by the log_interval? $\endgroup$ Jan 8, 2023 at 15:24
  • $\begingroup$ have no idea @MagedSaeed $\endgroup$
    – Faruk
    Jan 21, 2023 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.