I am wondering the calculation of perplexity
of a language model
which is based on character level LSTM model
. I got the code from kaggle and edited a bit for my problem but not the training way. I have added some other stuff to graph and save logs. However, as I am working on a language model
, I want to use perplexity
measuare to compare different results. In tensorflow
, I have done it via this answer and it was easy. I have looked for a way doing it in PyTorch
and literally no related result on Google. I need some help, and it is really appreciated.
Here is the related code, I believe:
criterion = nn.CrossEntropyLoss()
# create training and validation data
val_idx = int(len(data)*(1-val_frac))
data, val_data = data[:val_idx], data[val_idx:]
if(train_on_gpu):
net.cuda()
counter = 0
n_chars = len(net.chars)
for e in range(epochs):
# initialize hidden state
h = net.init_hidden(batch_size)
for x, y in get_batches(data, batch_size, seq_length):
counter += 1
# One-hot encode our data and make them Torch tensors
x = one_hot_encode(x, n_chars)
inputs, targets = torch.from_numpy(x), torch.from_numpy(y)
if(train_on_gpu):
inputs, targets = inputs.cuda(), targets.cuda()
# Creating new variables for the hidden state, otherwise
# we'd backprop through the entire training history
h = tuple([each.data for each in h])
# zero accumulated gradients
net.zero_grad()
# get the output from the model
output, h = net(inputs, h)
# calculate the loss and perform backprop
loss = criterion(output, targets.view(batch_size*seq_length))