7
$\begingroup$

Several neural network libraries such as tensorflow and pytorch offer an Embedding layer. Having implemented word2vec in the past, I understand the reasoning behind wanting a lower dimensional representation.

However, it would seem the embedding layer is just a linear layer. All other things being equal, would an embedding layer not just learn the same weights as the equivalent linear layer? If so, then what are the advantages of using an embedding layer?

In the case of word2vec, the lower dimensional representation can be used for other tasks (the infamous king/queen example). However, if your embedding layer will never be used for alternative tasks, what would be its purpose?

$\endgroup$
7
$\begingroup$

At theoretical level, the embedding layer is a linear layer, there is not any difference at all. However, in practice, if you are building a deep learning software, you have to make a difference among them. This is because it does not make sense to apply an embedding layer using traditional matrix multiplication, as the input matrix is very sparse. For this reason, it is faster to do a look-up, although in terms of theory it is equivalent to doing a matrix multiplication.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Can you please elaborate on the part of look-up ? I understand that doing matrix multiplation using one-hot-vector would be very time consuming bu then how it this problem solved using the embedding layer ? Basically how does embedding layer remains differentiable and still provides lookup functionality at scale ? $\endgroup$ – sahu Mar 27 '19 at 6:20
3
$\begingroup$

The embedding layer maps your vocabulary index input to a dense vector, so it acts as lookup layer and (if set to trainable) will be influenced on some weights only, by the words occurring in a batch of training data. Having a linear layer, it would be sequentially trained by all the data batches and would not provide a lookup functionality (each word given to the input would share the same weights).

Also, you're right considering word2vec differently. When using a custom trainable embedding layer, the dense vectors will be optimized (by SGD) for the task you are considering whereas models like word2vec act like language modeling and find a semantic optimum representation in the embeddings.

Therefore, depending on data size, the representation found during training might be better for your task than the one found by a neutral word2vec or other model.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.