I'm training a word2vec model based on our custom data. Due to the size of data and the resources available to me, I'll use pyspark to train spark's built-in word2vec model.

However, I'm curious to hear what are the best practices around training more effective, useful models?

For example, I'm currently not converting sentences to lower case, since in our scenario, capitalized words can have specific meaning.

I'm also not removing punctuation. Is that something practitioners find useful? Should I remove stop words or use TF-IDF to remove common words?

Is there a rule of thumb for the number of dimensions for a given amount of data. Is there a heuristic around the amount of data (in terms of gigabytes) I should feed this model?

I will be feeding this model all scraped text from our public websites, internal chat session and a few other sources. The output will be used for finding trends in chats, topic modeling, various types of classification.

Look forward to hearing about other people's experiences. I'll also be happy to review any other resources: papers, blog posts, youtube videos, etc.

On this site, this question from 2 year ago has some useful info, but I'm hoping for more suggestions: Word2Vec: Using pre-trained models


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.