I have a large compressed corpus, about 30gb in .txt.gz format. In raw format it can be used as input to word2vec like this:

data = gensim.models.word2vec.LineSentence(corpus)

This creates an iterator over the lines of the corpus. The next step is training:

model = gensim.models.Word2Vec(data)

I'd like to lemmatize and POS-tag the corpus before training. I am planning to use NLTK WordNetLemmatizer and NLTK POS-tagger.

How should I do this in a pipeline?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.