Why do we need 2 matrices for word2vec or GloVe

Question

Word2vec and GloVe are the two most known words embedding methods. Many works pointed that these two models are actually very close to each other and that under some assumptions, they perform a matrix factorization of the ppmi of the co-occurrences of the words in the corpus.

Still, I can't understand why we actually need two matrices (and not one) for these models. Couldn't we use the same one for U and V ? Is it a problem with the gradient descent or is there another reason ?

Someone told me it might be because the embeddings u and v of one word should be far enough to represent the fact that a word rarely appears in its own context. But it is not clear to me.

Answer 1

Might not be the answer you are seeking, but I'll still have a go:

First, quick review of word2Vec, assume we are using skip gram.

A typical Word2Vec train-able model consists of 1 input layer (for example, 10 000 long one-hot vector), a hidden layer (for example 300 neurons), an output (10 000 long one-hot vector)

• Input: 10 000
• Hidden: 300
• Output: 10 000

There is a matrix E between Input-Hidden, describing the weights to make your one-hot into an embedding. The matrix is special because each column (or rows, depending on your preferred notation) represents pre-activations in those 300 neurons - a response to a corresponding incoming 1-hot vector.

You don't need to perform any activation on these 300 neurons and can use their values straight away as an embedding in any future task.

---

However, simply squeezing a one-hot into a 300-dimensional representation isn't enough - it must have a meaning. And we ensure this meaning is correct using an additional second matrix - which connects Hidden to Output

We don't want to activate a hidden layer because activation-function won't be needed during runtime, however, in that case we will need a second matrix, going from Hidden to Output.

This second matrix will make an entirely different one-hot from your embeding. Such a one-hot will represent a most likely word to be nearby (contextually) of your original one-hot. In other words, this output won't be your original one-hot.

That's why a second matrix is needed. At the output, we perform a softmax, like in a classification problem.

This allows us to express a relation "word"-->embedding-->"context-neighbor-word"

Now, backpropagation can be done, to correct the Input-Hidden weights (Your first matrix E) - these are the weights we really care about. That's because Matrix E can be used during Runtime (I think), perhaps being plugged as a first fully-connected layer into some Recurrent Neural Net.

In that case you wouldn't use this:

You don't need to perform any activation on these 300 neurons and can use their values straight away as an embedding in any future task

but instead, you would just grab the appropriate column (or row, depending on your preferred notation) from that matrix, during Runtime. The benefit is that this way you get a very cheaply pre-trained fully-connected layer, designed to work with one-hots. Usually, first layers would be longest to train, due to the vanishing-gradient issue.

---

Why a second matrix is needed during training:

Once again, recall, there is no activation at the hidden layer.

We can dictate the network what "one-hot" must have been created in response to your original "input one-hot", and can punish network if it fails to generate a correct answer.

We cannot put softmax directly after hidden layer, because we are interested in summoning a mechanism to convert into an embedding. That's already a responsibility of a first matrix E. Thus, we need an extra step (an extra matrix) that will give us enough room to now form a conclusion at the output layer about a different but similar (context-wise) neighbor-word

During the runtime you throw away the second matrix. But don't delete it permanently in case if you need to come back and continue training your model.

Answer 2

why we actually need two matrices (and not one) for these models. Couldn't we use the same one for U and V?

In principle, you are right, we can. But we don't want to, since the increase in the number of parameters is beneficial in practice, but how about the meaning of the vectors?

What the papers say?

Word2vec: there is only one mention of input/output vectors just to introduce the variables.

GloVe: "the distinction between a word and a context word is arbitrary and that we are free to exchange the two roles".

Weak justification:

Vector pairs (word, context), also named (target, source) or (output, input), are used to reflect two different roles that a single word would play. However, since a word would play both "target" and "context" roles in the same sentence the justification is weak. For example, in sentence "I love deep learning", upon $P(deep | learning)$, $deep$ is a target, but next upon $P(learning | deep)$, it is a context, despite the fact that $deep$ has the same semantics in the whole sentence. We can conclude that this separation is not backed by the data.

So in the case of word embedding, this separation is just a story for the performance boost we get by increasing the number of parameters, nothing deeper whatsoever.

Solid justification:

However, in a task like node embedding in directed networks, the justification is solid because the roles are reflected at data level. For example, for a link $(a, b)$, $a$ is the source which is semantically different than being at the receiving end of the link $b$. A source node never plays the "target" role unless it is a target in another link. Thus you can expect to have a meaningful separation of semantics by using target (source) representations for target (source) nodes. This justification fails for undirected graphs the same as word embedding.

In some papers, authors even opt for four representations based on additional roles that an entity plays at data level, which is semantically justifiable and further boosts the performance.

Conclusion:

The whole premise is "if increase in the number of parameters pays off, do it. If it is semantically justifiable, even better."