0
$\begingroup$

I am wondering the calculation of perplexity of a language model which is based on character level LSTM model. I got the code from kaggle and edited a bit for my problem but not the training way. I have added some other stuff to graph and save logs. However, as I am working on a language model, I want to use perplexity measuare to compare different results. In tensorflow, I have done it via this answer and it was easy. I have looked for a way doing it in PyTorch and literally no related result on Google. I need some help, and it is really appreciated.

Here is the related code, I believe:

criterion = nn.CrossEntropyLoss()

# create training and validation data
val_idx = int(len(data)*(1-val_frac))
data, val_data = data[:val_idx], data[val_idx:]

if(train_on_gpu):
    net.cuda()

counter = 0
n_chars = len(net.chars)
for e in range(epochs):
    # initialize hidden state
    h = net.init_hidden(batch_size)

    for x, y in get_batches(data, batch_size, seq_length):
        counter += 1

        # One-hot encode our data and make them Torch tensors
        x = one_hot_encode(x, n_chars)
        inputs, targets = torch.from_numpy(x), torch.from_numpy(y)

        if(train_on_gpu):
            inputs, targets = inputs.cuda(), targets.cuda()

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients
        net.zero_grad()

        # get the output from the model
        output, h = net(inputs, h)

        # calculate the loss and perform backprop
        loss = criterion(output, targets.view(batch_size*seq_length))
$\endgroup$
3
$\begingroup$

I was surfing around at PyTorch's website and found a calculation of perplexity. You can examine how they calculated it as ppl as follows:

criterion = nn.CrossEntropyLoss()
total_loss = 0.
...
for batch, i in enumerate(range(0, train_data.size(0) - 1, bptt)):
    ...
    loss = criterion(output.view(-1, ntokens), targets)
    loss.backward()
    total_loss += loss.item()
    log_interval = 200
    if batch % log_interval == 0 and batch > 0:
        cur_loss = total_loss / log_interval
        ...
        print('ppl {:8.2f}'.format(math.exp(cur_loss)))
        ...

As @SpiderRico reminded, I got it from this link

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Including source of this code for sake of the completeness. github.com/pytorch/examples/tree/master/word_language_model (I assume this is where the OP got the code). $\endgroup$ – SpiderRico Mar 9 at 5:25
  • $\begingroup$ @SpiderRico Yes sir, you are correct. I have forgotten to add the link. Thanks for reminding. $\endgroup$ – Faruk Mar 9 at 10:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.