5
$\begingroup$

I used the gensim word2vec package and Keras Embedding layer for various different projects. Then I realize they seem to do the same thing, they all try to convert a word into a feature vector.

Am I understanding this properly? What exactly is the difference between these two methods?

Thanks!

$\endgroup$
3
$\begingroup$

Yep, you're right! As you know, it's difficult for machine learning models to use natural language directly, so it helps to transform words into some meaningful numeric representation. This process is called word embedding, and finding word embeddings is the task of the keras Embedding layer.

Ideally, word embeddings will be semantically meaningful, so that relationships between words are preserved in the embedding space. Word2Vec is a particular "brand" of word embedding algorithm that seeks to embed words such that words often found in similar context are located near one another in the embedding space. The technical details are described in this paper.

The generic keras Embedding layer also creates word embeddings, but the mechanism is a bit different than Word2Vec. Like any other layer, it is parameterized by a set of weights. The weights are randomly-initialized, then updated during training using the back-propagation algorithm. So, the resultant word embeddings are guided by your loss function.

To summarize, both Word2Vec and keras Embedding convert words (or word indices) to a hopefully meaningful numeric representation. Word2Vec is an unsupervised method that seeks to place words with similar context close together in the embedding space. Keras Embedding is a supervised method that finds custom embeddings while training your model.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks zachdj! Would you please elaborate a bit more why Keras Embedding is a supervised method? What is the ground truth in this case? $\endgroup$ – Edamame Oct 11 '19 at 20:42
  • 1
    $\begingroup$ Actually, I kind of regret saying that now. Keras embedding is not necessarily supervised or unsupervised - it's just a generic embedding layer that's trained like the rest of your model. $\endgroup$ – zachdj Oct 12 '19 at 3:16
  • $\begingroup$ It is an unsupervised learning. There is no labeling but since it is a seq2seq model, the labels are the actual data. $\endgroup$ – user702846 Mar 9 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.