1
$\begingroup$

I didn't find any function in nltk to calculate the perplexity.

There are some codes I found:

def calculate_bigram_perplexity(model, sentences):
    number_of_bigrams = model.corpus_length #calculate_number_of_bigrams(sentences)
    bigram_sentence_probability_log_sum = 0
    print("num of bigrams", number_of_bigrams)
    for sentence in sentences:
        try:
            bigram_sentence_probability_log_sum -= math.log(model.calculate_bigram_sentence_probability(sentence), 2)
        except:
            bigram_sentence_probability_log_sum -= float('-inf')
    x = math.pow(2, bigram_sentence_probability_log_sum / number_of_bigrams)
    y = math.pow(2, nltk.probability.entropy(model.prob_dist))
    print(f"x = {x} and y = {y}")
    return y

in the code above x is the output of the function, however, I also calculated it from another method:

y = math.pow(2, nltk.probability.entropy(model.prob_dist))

My question is that which of these methods are correct, because they give me different results. Moreover, my results for bigram and unigram differs:

== TEST PERPLEXITY == 
unigram perplxity:
x = 447.0296119273938 and y = 553.6911988953756
unigram:  553.6911988953756
=============
num of bigrams 23102
x = 1.530813112747101 and y = 7661.285234275603
bigram perplxity:  7661.285234275603

I expected to see lower perplexity for bigram, but it's much higher, what could be the problem of calculation? Please note that I process a text involving multiple sentences... could they be because of sparse data, because I just tested them on one text.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.