I am trying to train wor2vec embeddings on tweets. I defined the sentence class as follows:
def tokenize_tweets(): for line in codecs.open('../data/sample_tweets.txt', encoding='utf-8'): tweet_text = ' '.join([token for token in tknz.tokenize(line) if token not in stopwords.words('english')]) try: mod_text = tokenize(tweet_text) tokens = tknz.tokenize(mod_text) if len(tokens) > 0: yield tknz.tokenize(mod_text) else: yield ['NULL'] except UnicodeEncodeError as e: yield ['<NULL>']
Voacb. building from this class runs fine. But when I try running the train method, I am getting the following errors:
ValueError: You must specify either total_examples or total_words, for proper alpha and progress calculations. The usual value is total_examples=model.corpus_count.
Not sure what is wrong with it.