When i come to know that gensim is useful library for topic modeling, I tried it on my huge amount of document. It works well only if the dictionary size is to be fix. In my case i have each and every token is important so i would have huge amount of token at that time gensim algo like(LDA,LSA,etc) fails and get out of memory execution issue. Actually they written that there is no limit for amount of document but internally they keep the dictionary size fixed that's why if new word will come and exceed the limit than it starts to truncate it. I dont want to truncate my token as possible. So is there any solution for that can resolve my issue of topic modeling on big data. I have resource limitation.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Use filter_extremes to Filter out tokens that appear in less than no_below documents (absolute number) or more than no_above documents (fraction of total corpus size, not absolute number).
keep only the first keep_n most frequent tokens (or keep all if None).
After the pruning, shrink resulting gaps in word ids.
