Word2vec and GloVe are the two most known words embedding methods. Many works pointed that these two models are actually very close to each other and that under some assumptions, they perform a matrix factorization of the ppmi of the co-occurrences of the words in the corpus.

Still, I can't understand why we actually need two matrices (and not one) for these models. Couldn't we use the same one for U and V ? Is it a problem with the gradient descent or is there another reason ?

Someone told me it might be because the embeddings u and v of one word should be far enough to represent the fact that a word rarely appears in its own context. But it is not clear to me.

  • $\begingroup$ Their dimensions are different; how could you use the same matrix? But they do live in the same latent space. $\endgroup$ – Emre Mar 13 '18 at 16:37
  • 1
    $\begingroup$ They do have the same dimensions (number of words in the vocabulary x embedding dimension) $\endgroup$ – Robin Mar 13 '18 at 16:45

Might not be the answer you are seeking, but I'll still have a go:

First, quick review of word2Vec, assume we are using skip gram.

A typical Word2Vec train-able model consists of 1 input layer (for example, 10 000 long one-hot vector), a hidden layer (for example 300 neurons), an output (10 000 long one-hot vector)

  • Input: 10 000
  • Hidden: 300
  • Output: 10 000

There is a matrix E between Input-Hidden, describing the weights to make your one-hot into an embedding. The matrix is special because each column (or rows, depending on your preferred notation) represents pre-activations in those 300 neurons - a response to a corresponding incoming 1-hot vector.

You don't need to perform any activation on these 300 neurons and can use their values straight away as an embedding in any future task.

However, simply squeezing a one-hot into a 300-dimensional representation isn't enough - it must have a meaning. And we ensure this meaning is correct using an additional second matrix - which connects Hidden to Output

We don't want to activate a hidden layer because activation-function won't be needed during runtime, however, in that case we will need a second matrix, going from Hidden to Output.

This second matrix will make an entirely different one-hot from your embeding. Such a one-hot will represent a most likely word to be nearby (contextually) of your original one-hot. In other words, this output won't be your original one-hot.

That's why a second matrix is needed. At the output, we perform a softmax, like in a classification problem.

This allows us to express a relation "word"-->embedding-->"context-neighbor-word"

Now, backpropagation can be done, to correct the Input-Hidden weights (Your first matrix E) - these are the weights we really care about. That's because Matrix E can be used during Runtime (I think), perhaps being plugged as a first fully-connected layer into some Recurrent Neural Net.

In that case you wouldn't use this:

You don't need to perform any activation on these 300 neurons and can use their values straight away as an embedding in any future task

but instead, you would just grab the appropriate column (or row, depending on your preferred notation) from that matrix, during Runtime. The benefit is that this way you get a very cheaply pre-trained fully-connected layer, designed to work with one-hots. Usually, first layers would be longest to train, due to the vanishing-gradient issue.

Why a second matrix is needed during training:

Once again, recall, there is no activation at the hidden layer.

We can dictate the network what "one-hot" must have been created in response to your original "input one-hot", and can punish network if it fails to generate a correct answer.

We cannot put softmax directly after hidden layer, because we are interested in summoning a mechanism to convert into an embedding. That's already a responsibility of a first matrix E. Thus, we need an extra step (an extra matrix) that will give us enough room to now form a conclusion at the output layer about a different but similar (context-wise) neighbor-word

During the runtime you throw away the second matrix. But don't delete it permanently in case if you need to come back and continue training your model.

| improve this answer | |
  • $\begingroup$ For word2vec, in practice, we perform a gradient descent from the context output layer through the embeddings to the input word layer. But nothing prevent you from having the same matrix for what you call the extra matrix. Moreover, when computing the embeddings, you can actually sum (or concat) these two matrices and some papers report you get even better embeddings. I agree that if you consider the neural model, all this makes sense, but there must be an explanation why we need them to be two different entities. $\endgroup$ – Robin Mar 19 '18 at 13:40
  • $\begingroup$ In other word, you could buold you model like this: input layer * embedding matrix => embeddings / embeddings * transpose of bedding matrix => output layer $\endgroup$ – Robin Mar 19 '18 at 13:42
  • $\begingroup$ In that case I would be interested in this info, if that's indeed possible. I suggest to email all 5 authors separately, and explaining the issue, one of them will surely reply. The authors from here Please tell the outcome if the approach is viable $\endgroup$ – Kari Mar 19 '18 at 14:24
  • $\begingroup$ Maybe it's related to the cost of computing inverse matrix (on every iteration) - it would be quite large I suppose $\endgroup$ – Kari Mar 21 '18 at 19:02
  • 1
    $\begingroup$ Transpose is not inverse. The only non-standard thing in what debzsud proposes would be weight sharing between layers so you'd need to calculate backpropagation with shared weights - which should be doable. I suppose the only question if twice fewer parameters are enough to solve the same problem. It should not be too hard to implement and compare youself. $\endgroup$ – Valentas Mar 25 '18 at 8:54

why we actually need two matrices (and not one) for these models. Couldn't we use the same one for U and V?

In principle, you are right, we can. But we don't want to, since the increase in the number of parameters is beneficial in practice, but how about the meaning of the vectors?

What the papers say?

Word2vec: there is only one mention of input/output vectors just to introduce the variables.

GloVe: "the distinction between a word and a context word is arbitrary and that we are free to exchange the two roles".

Weak justification:

Vector pairs (word, context), also named (target, source) or (output, input), are used to reflect two different roles that a single word would play. However, since a word would play both "target" and "context" roles in the same sentence the justification is weak. For example, in sentence "I love deep learning", upon $P(deep | learning)$, $deep$ is a target, but next upon $P(learning | deep)$, it is a context, despite the fact that $deep$ has the same semantics in the whole sentence. We can conclude that this separation is not backed by the data.

So in the case of word embedding, this separation is just a story for the performance boost we get by increasing the number of parameters, nothing deeper whatsoever.

Solid justification:

However, in a task like node embedding in directed networks, the justification is solid because the roles are reflected at data level. For example, for a link $(a, b)$, $a$ is the source which is semantically different than being at the receiving end of the link $b$. A source node never plays the "target" role unless it is a target in another link. Thus you can expect to have a meaningful separation of semantics by using target (source) representations for target (source) nodes. This justification fails for undirected graphs the same as word embedding.

In some papers, authors even opt for four representations based on additional roles that an entity plays at data level, which is semantically justifiable and further boosts the performance.


The whole premise is "if increase in the number of parameters pays off, do it. If it is semantically justifiable, even better."

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.