Gensim word2vec training error on tweets

Question

I am trying to train wor2vec embeddings on tweets. I defined the sentence class as follows:

def tokenize_tweets():
    for line in codecs.open('../data/sample_tweets.txt', encoding='utf-8'):
        tweet_text = ' '.join([token for token in tknz.tokenize(line) if token not in stopwords.words('english')])
        try:
            mod_text = tokenize(tweet_text)
            tokens = tknz.tokenize(mod_text)
            if len(tokens) > 0:
                yield tknz.tokenize(mod_text)
            else:
                yield ['NULL']
        except UnicodeEncodeError as e:
            yield ['<NULL>']

Voacb. building from this class runs fine. But when I try running the train method, I am getting the following errors:

ValueError: You must specify either total_examples or total_words, for proper alpha and progress calculations. The usual value is total_examples=model.corpus_count.

Not sure what is wrong with it.

Answer 1

In newer W2V version it is not enough just to write:

model_name.train(sentences)

You have to write inside word_count or total_examples.

For example, I write:

model_name.train(sentences, total_examples = token_count, epochs = model_name.iter )

where token_count = sum([len(sentence) for sentence in sentences]). And here is how I get sentences:

sentences = []
for raw_sentence in raw_sentences:

    if len(raw_sentence) > 0:
        sentences.append(sentence_to_wordlist(raw_sentence))

More documentation here.

Answer 2

Please refer this github repo for solution - https://github.com/RaRe-Technologies/gensim/issues/1284