# How to calculate perplexity in PyTorch?

I am wondering the calculation of perplexity of a language model which is based on character level LSTM model. I got the code from kaggle and edited a bit for my problem but not the training way. I have added some other stuff to graph and save logs. However, as I am working on a language model, I want to use perplexity measuare to compare different results. In tensorflow, I have done it via this answer and it was easy. I have looked for a way doing it in PyTorch and literally no related result on Google. I need some help, and it is really appreciated.

Here is the related code, I believe:

criterion = nn.CrossEntropyLoss()

# create training and validation data
val_idx = int(len(data)*(1-val_frac))
data, val_data = data[:val_idx], data[val_idx:]

if(train_on_gpu):
net.cuda()

counter = 0
n_chars = len(net.chars)
for e in range(epochs):
# initialize hidden state
h = net.init_hidden(batch_size)

for x, y in get_batches(data, batch_size, seq_length):
counter += 1

# One-hot encode our data and make them Torch tensors
x = one_hot_encode(x, n_chars)
inputs, targets = torch.from_numpy(x), torch.from_numpy(y)

if(train_on_gpu):
inputs, targets = inputs.cuda(), targets.cuda()

# Creating new variables for the hidden state, otherwise
# we'd backprop through the entire training history
h = tuple([each.data for each in h])

# get the output from the model
output, h = net(inputs, h)

# calculate the loss and perform backprop
loss = criterion(output, targets.view(batch_size*seq_length))


I was surfing around at PyTorch's website and found a calculation of perplexity. You can examine how they calculated it as ppl as follows:

criterion = nn.CrossEntropyLoss()
total_loss = 0.
...
for batch, i in enumerate(range(0, train_data.size(0) - 1, bptt)):
...
loss = criterion(output.view(-1, ntokens), targets)
loss.backward()
total_loss += loss.item()
log_interval = 200
if batch % log_interval == 0 and batch > 0:
cur_loss = total_loss / log_interval
...
print('ppl {:8.2f}'.format(math.exp(cur_loss)))
...


As @SpiderRico reminded, I got it from this link

• Including source of this code for sake of the completeness. github.com/pytorch/examples/tree/master/word_language_model (I assume this is where the OP got the code). – SpiderRico Mar 9 at 5:25
• @SpiderRico Yes sir, you are correct. I have forgotten to add the link. Thanks for reminding. – Faruk Mar 9 at 10:37