Given one language ngram model, how do I compare likelihoods of two texts of different length?

Question

Let's say I have conditional probabilities estimates for N-grams and I want to find out which of the two sequences of different length 'looks more natural' in terms of the given model.

How does one do this? Is it something to do with perplexity?

Would also be glad if someone posted a reference to a reasonable Python implementation of the suggested method [a code snippet or a link to nltk/spacy/whatever method].

Thanks in advance.

Answer 1

I didn't find any good explanations or papers on this topic, other than things about category based models and multigrams (parts of words). So, I came up with one myself. I'm using Java, but here's the code for the length normalization translated to Python. Also, this is assuming that you're adding the log probabilities for each word together, like KenLM does for the total sentence (or phrase) score.

# Get likelihood normalized by length and number of words.
# Since the likelihoods of each word are added together, phrases with more words tend to 
# have a lower probability.
def getNormLikelihood(phrase):
    score = getLikelihood(phrase) # The normal score
    numWords = phrase.count(' ')+1 # The number of words in the phrase.
    # Get the absolute difference between the 
    #   average word length of all words and the average word length of the phrase.
    # Then, weight that weight by the reciprocal of the normal score, such that the 
    #   lenWeight is less if the overall score is less.
    # Intuitively, this means that we care less about normalizing by word length the more 
    #   common the phrase is.
    # This effectively solves the problem of giving a greater likelihood for long words 
    #   that aren't even in the language model (and are given the default "<unk>" score).
    lenWeight = abs(len(phrase) / numWords - avgLen) * (-1/score)
    return score / (1/lenWeight)

How you want to use this normalized score, and how effective it is, depends on your application. In my case (splitting words), I used this normalized score to score each splitting option, but then also added (logarithm) the normal score of the option with context, which you could consider the prior for the option (in Naive Bayes terms). I then added an overall prior (manually picked, but could be tuned) to the original, unsplit option, for the probability of needing to split a word. The normalized score greatly helped, and with these probabilities together, I was able to get 100% accuracy for my test cases.