The topic modelling library Gensim offers the ability to stream a large document instead of storing it in memory.
Streaming is possible for the stage of converting the corpus to BOW, but the dictionary must have been created first and apparently this requires loading the full corpus:
from gensim import corpora dictionary = corpora.Dictionary(a_huge_corpus)
Is there a way to create the dictionary without loading the whole corpus at once?
ps: this is my first question on the site, I'm a beginner ;)