The topic modelling library Gensim offers the ability to stream a large document instead of storing it in memory.

Streaming is possible for the stage of converting the corpus to BOW, but the dictionary must have been created first and apparently this requires loading the full corpus:

from gensim import corpora

dictionary = corpora.Dictionary(a_huge_corpus)

Is there a way to create the dictionary without loading the whole corpus at once?

ps: this is my first question on the site, I'm a beginner ;)


1 Answer 1


Not completely sure if the issue having the full corpus in memory (i.e. the a_huge_corpus variable) or having the full Dictionary in memory, but in the case of the first one it seems that the add_documents method useful to add part of the corpus at a time:

from gensim.corpora import Dictionary

corpus = [
    ["this", "is", "a", "random", "sentence"],
    ["this", "is", "another", "random", "sentence"],
    ["and", "another", "one"],
    ["this", "is", "sparta"],
    ["just", "joking"]
dct = Dictionary()
print(dct == Dictionary(corpus))
# True

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.