2
$\begingroup$

I am trying to train wor2vec embeddings on tweets. I defined the sentence class as follows:

def tokenize_tweets():
    for line in codecs.open('../data/sample_tweets.txt', encoding='utf-8'):
        tweet_text = ' '.join([token for token in tknz.tokenize(line) if token not in stopwords.words('english')])
        try:
            mod_text = tokenize(tweet_text)
            tokens = tknz.tokenize(mod_text)
            if len(tokens) > 0:
                yield tknz.tokenize(mod_text)
            else:
                yield ['NULL']
        except UnicodeEncodeError as e:
            yield ['<NULL>']

Voacb. building from this class runs fine. But when I try running the train method, I am getting the following errors:

ValueError: You must specify either total_examples or total_words, for proper alpha and progress calculations. The usual value is total_examples=model.corpus_count.

Not sure what is wrong with it.

$\endgroup$
2
$\begingroup$

In newer W2V version it is not enough just to write:

model_name.train(sentences)

You have to write inside word_count or total_examples.

For example, I write:

model_name.train(sentences, total_examples = token_count, epochs = model_name.iter )

where token_count = sum([len(sentence) for sentence in sentences]). And here is how I get sentences:

sentences = []
for raw_sentence in raw_sentences:

    if len(raw_sentence) > 0:
        sentences.append(sentence_to_wordlist(raw_sentence))

More documentation here.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Please refer this github repo for solution - https://github.com/RaRe-Technologies/gensim/issues/1284

| improve this answer | |
$\endgroup$
  • $\begingroup$ Welcome, new user. The rules of StackExchange require you to elaborate on the answers. It is normally not allowed to link a webpage and limit yourself to a "check this out". $\endgroup$ – Leevo Aug 21 '19 at 7:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.