I'm trying to compare texts (read: books) using KL divergence of N-gram usage frequency.
first I have to calculate the frequency of N-grams, and I see (perhaps unsurprisingly) that many of the words used are "rare" (for example, they appear <10 times in the text). I wish to replace these rare words with tokens ("UNK", for unknown word).
Let's say I am comparing 10 texts. At what point should I count the rare words to discover which can be considered rare? I can think of three options:
- Count words for each text separately. If "Boat" is a word is rare in the first text and frequent in second one, it is replaced with UNK only in the first text, and not in the second one.
- Count words for each comparison. For 10 texts I'm carrying out 45 (two-sided) comparisons. If I choose this option, I will not replace the word "Boat" in either text when comparing the first and second texts. I will however replace it with UNK when comparing the first and third texts (assume that "Boat" is rare in the third text as well).
- Count words for the entire collection. Using this option, "Boat" is never replaced in any of the texts, due to it being a non-rare word in one of the texts.
What option would serve me better in finding meaningful N-gram representations of texts?
Many thanks.