What is the standard output of the GloVe algorithm?

Question

When I look at the loss function of the GloVe algorithm for generating word vectors, I see that $w$ and $\tilde{w}$ are symmetric:

$$ J=\sum_{i,j=1}^Vf(X_{ij})(w_i^T\tilde{w}_j+b_i+b_j-logX_{ij})^2 $$

(eq. 8 from https://nlp.stanford.edu/pubs/glove.pdf)

However, the authors never seem to describe whether $w$ and $\tilde{w}$ are both used to generate the embedding, or only one? When I watch Andrew Ng describe the algorithm (https://www.youtube.com/watch?v=EHXqgQNu-Iw), he suggests averaging the two, but when I try to read the original c-implementation of GloVe (https://github.com/stanfordnlp/GloVe/blob/master/src/glove.c), it seems that they implement both saving $w$, and their concatenation, depending on user settings. What do people usually do?

Answer 1

If you are using the embeddings in a downstream task, better performance is often achieved by averaging or summing them (the factor of 2 makes no difference). Concatenating them is fine also as it preserves that same information, i.e. the downstream model can effectively average them (of course it preserves more info, but it is unknown if that is beneficial). Averaging/summing is obviously more memory efficient than concatenating.

An intuition for why averaging is useful is in this paper, which relates mainly to word2vec but both models learn similar statistics (which is ultimately what these embedding capture). This paper examines the effect of adding the embeddings and offers a different explanation.