2
$\begingroup$

So I am learning Word2Vec for the first time and my question is quite basic: How to know what approach to use? I.e, Word2Vec in Tensorflow or Word2Vec trained with Gensim ?

In what cases would implementing it through the more manual first approach be useful vs. the second one? If there is already an easier way to train a word2vec model using gensim, why is that not used always?

Furthermore, what is the benefit in using a pre-trained model like the Google News dataset? What happens when there are words that are not included in the news dataset?

Sorry if this question is basic, I just want to get a clearer grasp of the overall picture.

$\endgroup$
3
$\begingroup$

The advantage of using pre-trained vectors is being able to inject knowledge from a larger corpus than you might have access to: word2vec has a vocabulary of 3 million words and phrases trained on the google news dataset comprising ~100 billion tokens, and there's no cost to you in training time.

In addition, they are fast and easy to use, just load the embeddings and look them up. It's straightforward to substitute different sets of pre-trained vectors (fastText, GloVe etc) as one might be more suited to a particular use case.

However, when your vocabulary does not have an entry in word2vec, by default you'll end up with a null entry in your embedding layer (depending on how you handle it). You'll need to consider the scale/impact and how to address it (keep/discard/consider online training). As yazhi says a decision must be made about how to handle out of vocabulary words.

The advantage of learning word vectors from your own corpus is that they would be derived from your dataset, so if you have reason to believe that the composition of your data is significantly different from the corpus used for the pre-trained vectors then that may result in better downstream performance. However, that comes at a cost in time taken to train your own vector representations.

$\endgroup$
1
$\begingroup$

Tensorflow has implementations for a pool of machine learning algorithms, so it should be comfortable if your application needs to build something on top of word2vec. Gensim is mainly intended for topic modelling techniques, but pretty robust as its their main work.

If you want to get a clear grasp of how the algorithm works, then implementing manually makes sense. Else, just go with one of the implementations.

Google word2vec model is pretty good and covers most of the English words. Use it, if you do not have computational power or time to train a model. Manual training gives you the freedom of choosing domain-specific dataset, window size, cbow or sg, length of vector. If there are Out of vocabulary(OOV) words, it will throw an error.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.