0
$\begingroup$

When I look at the loss function of the GloVe algorithm for generating word vectors, I see that $w$ and $\tilde{w}$ are symmetric:

$$ J=\sum_{i,j=1}^Vf(X_{ij})(w_i^T\tilde{w}_j+b_i+b_j-logX_{ij})^2 $$

(eq. 8 from https://nlp.stanford.edu/pubs/glove.pdf)

However, the authors never seem to describe whether $w$ and $\tilde{w}$ are both used to generate the embedding, or only one? When I watch Andrew Ng describe the algorithm (https://www.youtube.com/watch?v=EHXqgQNu-Iw), he suggests averaging the two, but when I try to read the original c-implementation of GloVe (https://github.com/stanfordnlp/GloVe/blob/master/src/glove.c), it seems that they implement both saving $w$, and their concatenation, depending on user settings. What do people usually do?

$\endgroup$

1 Answer 1

1
$\begingroup$

If you are using the embeddings in a downstream task, better performance is often achieved by averaging or summing them (the factor of 2 makes no difference). Concatenating them is fine also as it preserves that same information, i.e. the downstream model can effectively average them (of course it preserves more info, but it is unknown if that is beneficial). Averaging/summing is obviously more memory efficient than concatenating.

An intuition for why averaging is useful is in this paper, which relates mainly to word2vec but both models learn similar statistics (which is ultimately what these embedding capture). This paper examines the effect of adding the embeddings and offers a different explanation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.