I didn't find any function in nltk to calculate the perplexity.

There are some codes I found:

def calculate_bigram_perplexity(model, sentences):
    number_of_bigrams = model.corpus_length #calculate_number_of_bigrams(sentences)
    bigram_sentence_probability_log_sum = 0
    print("num of bigrams", number_of_bigrams)
    for sentence in sentences:
            bigram_sentence_probability_log_sum -= math.log(model.calculate_bigram_sentence_probability(sentence), 2)
            bigram_sentence_probability_log_sum -= float('-inf')
    x = math.pow(2, bigram_sentence_probability_log_sum / number_of_bigrams)
    y = math.pow(2, nltk.probability.entropy(model.prob_dist))
    print(f"x = {x} and y = {y}")
    return y

in the code above x is the output of the function, however, I also calculated it from another method:

y = math.pow(2, nltk.probability.entropy(model.prob_dist))

My question is that which of these methods are correct, because they give me different results. Moreover, my results for bigram and unigram differs:

unigram perplxity:
x = 447.0296119273938 and y = 553.6911988953756
unigram:  553.6911988953756
num of bigrams 23102
x = 1.530813112747101 and y = 7661.285234275603
bigram perplxity:  7661.285234275603

I expected to see lower perplexity for bigram, but it's much higher, what could be the problem of calculation? Please note that I process a text involving multiple sentences... could they be because of sparse data, because I just tested them on one text.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.