Does sum of embeddings make sense?

Question

Referring to the LightFM model from paper Metadata Embeddings for User and Item Cold-start Recommendations, the model tries to learn $d$-dimensional user and item feature embeddings $e_f^U$ and $e_f^I$ for each feature $f$ ($U$ is the set of users, $I$ is the set of items).

The latent representation of user $u$ is given by the sum of its features' latent vectors: $$ \mathbf{q_u} = \sum_{j\in{f_u}}\mathbf{e_j^U} $$

The same holds for item $i$: $$ \mathbf{p_i} = \sum_{j\in{f_i}}\mathbf{e_j^I} $$

Does it really make sense to sum the latent embeddings to represent a set of features (user or item)?

Answer 1

Thinking of a sum as an average, this model sounds reminiscent of continuous bag of words (CBOW) word embeddings (i.e. word2vec). In that context, the words in a sentence are used to predict a missing word using the average of embedded vectors (see this question). This method works pretty well for words, so it might make sense that you could extend it to other types of embeddings like user/item compatibility.

The average makes some intuitive sense when you think of comparisons. To place a user/item in embedding space, you can average the feature embeddings. If a user/item has different but similar features their average should put the user/item in a similar region of embedding space. A sum should work the same if the feature number is fixed.

An issue with a sum would arise if feature number isn't fixed. Users/items could appear different because their embedding vector is a different length, even if it has the same direction. Normalizing by feature number would fix this.

Another possible issue with using a sum rather than an average is that the user/item embedded vectors aren't comparable to the feature embedded vectors. This doesn't seem to matter for the author, because they don't try to compare the user/item vectors to the feature vectors directly.

TL;DR Yes, a sum might make sense to represent a set of features, but an average might be better.

Answer 2

Summing Embeddings can makes sense under certain conditions.

If you want to just do a naive euclidian distance based k-NN to find close samples, summing the vectors doesn't work because the tip of the vector gets moved away from the origin.

This is easy to visualize if you add the same vector to itself (or scale it with a factor of 2), there will be a new euclidian distance to it's un-summed version that is equal to its own magnitude.

But there are other metrics where this does make sense like the cosine distance.

As this only looks at the relative angle between the vectors, the magnitude of the vectors doesn't influence the result. So a scaled vector will still give you a cosine distance of 1 to it's unscaled variant.