0
$\begingroup$

I'm trying to compare texts (read: books) using KL divergence of N-gram usage frequency.

first I have to calculate the frequency of N-grams, and I see (perhaps unsurprisingly) that many of the words used are "rare" (for example, they appear <10 times in the text). I wish to replace these rare words with tokens ("UNK", for unknown word).

Let's say I am comparing 10 texts. At what point should I count the rare words to discover which can be considered rare? I can think of three options:

  1. Count words for each text separately. If "Boat" is a word is rare in the first text and frequent in second one, it is replaced with UNK only in the first text, and not in the second one.
  2. Count words for each comparison. For 10 texts I'm carrying out 45 (two-sided) comparisons. If I choose this option, I will not replace the word "Boat" in either text when comparing the first and second texts. I will however replace it with UNK when comparing the first and third texts (assume that "Boat" is rare in the third text as well).
  3. Count words for the entire collection. Using this option, "Boat" is never replaced in any of the texts, due to it being a non-rare word in one of the texts.

What option would serve me better in finding meaningful N-gram representations of texts?

Many thanks.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

On one side, for each compared pair of documents you would like the same word being represented the same way in both documents. That would be option 3. - 'global rare words'.

On the other side, option 1. and 2. are more easily calculated and scaled with subsequent documents.

I would try calculating rare words per document but skipping all n-grams with them - as these n-grams will also be rare.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So you say that simply skipping ngrams with rare words would be better than replacing the rare words with an 'UNK' token? $\endgroup$
    – Lafayette
    Jul 22, 2018 at 7:58
  • 1
    $\begingroup$ Sorry for silence, holiday break :) Yes, this is what I meant, here stackoverflow.com/questions/43885252/… is similar questions with interesting link about 'smoothing' probabilities in case of rare events. $\endgroup$
    – MkL
    Aug 7, 2018 at 18:38
1
$\begingroup$

Tf-idf (text frequency - Inverness document frequency) will highlight your interests.

https://en.m.wikipedia.org/wiki/Tf–idf

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you, I am familiar with TF-IDF. However in this case I do not wish to use that approach. $\endgroup$
    – Lafayette
    Jul 17, 2018 at 5:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.