Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them?

That is, to get results which say that, for example, the bigram "hello world" is much more common in corpus A than corpus B (ideally with some kind of measure of how much more common).

TF-IDF examples use a larger number of documents to highlight "important" words in each, but I am not sure if that would work in this case?


As far as I understood your question you can work with NLTK.

Have a look on the following functions:

  • Tokenize
  • Stemmers
  • pos_tags (if you wanna categorize by Verb, Noun etc. beforehand)

And to come to your question of similarity or relative word/phrase usage you have FreqDistfunction in NLTK. Using this Output you could set each occurance relative to the length of the list that is returned by tokenize


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.