I'm trying to compare texts (read: books) using KL divergence of N-gram usage frequency.

first I have to calculate the frequency of N-grams, and I see (perhaps unsurprisingly) that many of the words used are "rare" (for example, they appear <10 times in the text). I wish to replace these rare words with tokens ("UNK", for unknown word).

Let's say I am comparing 10 texts. At what point should I count the rare words to discover which can be considered rare? I can think of three options:

  1. Count words for each text separately. If "Boat" is a word is rare in the first text and frequent in second one, it is replaced with UNK only in the first text, and not in the second one.
  2. Count words for each comparison. For 10 texts I'm carrying out 45 (two-sided) comparisons. If I choose this option, I will not replace the word "Boat" in either text when comparing the first and second texts. I will however replace it with UNK when comparing the first and third texts (assume that "Boat" is rare in the third text as well).
  3. Count words for the entire collection. Using this option, "Boat" is never replaced in any of the texts, due to it being a non-rare word in one of the texts.

What option would serve me better in finding meaningful N-gram representations of texts?

Many thanks.


On one side, for each compared pair of documents you would like the same word being represented the same way in both documents. That would be option 3. - 'global rare words'.

On the other side, option 1. and 2. are more easily calculated and scaled with subsequent documents.

I would try calculating rare words per document but skipping all n-grams with them - as these n-grams will also be rare.

  • $\begingroup$ So you say that simply skipping ngrams with rare words would be better than replacing the rare words with an 'UNK' token? $\endgroup$ – Lafayette Jul 22 '18 at 7:58
  • 1
    $\begingroup$ Sorry for silence, holiday break :) Yes, this is what I meant, here stackoverflow.com/questions/43885252/… is similar questions with interesting link about 'smoothing' probabilities in case of rare events. $\endgroup$ – MkL Aug 7 '18 at 18:38

Tf-idf (text frequency - Inverness document frequency) will highlight your interests.


  • $\begingroup$ Thank you, I am familiar with TF-IDF. However in this case I do not wish to use that approach. $\endgroup$ – Lafayette Jul 17 '18 at 5:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.