Gensim: create a dictionary from a large corpus without loading it in RAM?

Question

The topic modelling library Gensim offers the ability to stream a large document instead of storing it in memory.

Streaming is possible for the stage of converting the corpus to BOW, but the dictionary must have been created first and apparently this requires loading the full corpus:

from gensim import corpora

dictionary = corpora.Dictionary(a_huge_corpus)

Is there a way to create the dictionary without loading the whole corpus at once?

ps: this is my first question on the site, I'm a beginner ;)

Answer 1

Not completely sure if the issue having the full corpus in memory (i.e. the a_huge_corpus variable) or having the full Dictionary in memory, but in the case of the first one it seems that the add_documents method useful to add part of the corpus at a time:

from gensim.corpora import Dictionary

corpus = [
    ["this", "is", "a", "random", "sentence"],
    ["this", "is", "another", "random", "sentence"],
    ["and", "another", "one"],
    ["this", "is", "sparta"],
    ["just", "joking"]
]
dct = Dictionary()
dct.add_documents(corpus[:3])
dct.add_documents(corpus[3:])
print(dct == Dictionary(corpus))
# True