# Gensim word2vec training error on tweets

I am trying to train wor2vec embeddings on tweets. I defined the sentence class as follows:

def tokenize_tweets():
for line in codecs.open('../data/sample_tweets.txt', encoding='utf-8'):
tweet_text = ' '.join([token for token in tknz.tokenize(line) if token not in stopwords.words('english')])
try:
mod_text = tokenize(tweet_text)
tokens = tknz.tokenize(mod_text)
if len(tokens) > 0:
yield tknz.tokenize(mod_text)
else:
yield ['NULL']
except UnicodeEncodeError as e:
yield ['<NULL>']


Voacb. building from this class runs fine. But when I try running the train method, I am getting the following errors:

ValueError: You must specify either total_examples or total_words, for proper alpha and progress calculations. The usual value is total_examples=model.corpus_count.


Not sure what is wrong with it.

In newer W2V version it is not enough just to write:

model_name.train(sentences)


You have to write inside word_count or total_examples.

For example, I write:

model_name.train(sentences, total_examples = token_count, epochs = model_name.iter )


where token_count = sum([len(sentence) for sentence in sentences]). And here is how I get sentences:

sentences = []
for raw_sentence in raw_sentences:

if len(raw_sentence) > 0:
sentences.append(sentence_to_wordlist(raw_sentence))


More documentation here.

Please refer this github repo for solution - https://github.com/RaRe-Technologies/gensim/issues/1284

• Welcome, new user. The rules of StackExchange require you to elaborate on the answers. It is normally not allowed to link a webpage and limit yourself to a "check this out". – Leevo Aug 21 '19 at 7:45