I have a large corpus of documents (multiple sentences per document), and am trying to find words that appear in some majority of documents (to then filter them out of a future analysis). Since I'm looking for both a term frequency and document frequency TF-IDF obviously comes to mind, however I don't think the TF-IDF score will quite get me what I'm looking for.

Basically I'd like to end up with a list of all words in my corpus, along with the percentage of documents that word appears in.

Is there a best practice here? I've looked through the NLTK and spaCy docs, but neither have anything quite like what I'm looking for.

  • $\begingroup$ "along with the percentage of documents that word appears in" thats the document frequency $\endgroup$ – Alejandro Celis Mar 18 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.