NLP Transformers: How to get a fixed sentences embedding vectors size?

Question

I'm loading a language model from torch hub (CamemBERT a French RoBERTa-based model) and using it do embed some sentences:

import torch
camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
camembert.eval()  # disable dropout (or leave in train mode to finetune)


def embed(sentence):
   tokens = camembert.encode(sentence)
   # Extract all layer's features (layer 0 is the embedding layer)
   all_layers = camembert.extract_features(tokens, return_all_hiddens=True)
   embeddings = all_layers[0]
   return embeddings

# Here we see that the shape of the embedding vector depends on the number of tokens in the sentence

u = embed("Bonjour, ça va ?")
u.shape # torch.Size([1, 7, 768])
v = embed("Salut, comment vas-tu ?")
v.shape # torch.Size([1, 9, 768])

Imagine now, I want to calculate the cosine distance between the vectors (tensors in our case) u and v :

cos = torch.nn.CosineSimilarity(dim=0)
cos(u, v) #will throw an error since the shape of `u` is different from the shape of `v``

I'm asking what is the best method to use in order to always get the same embedding shape for a sentence regardless the count of tokens?

=> The first solution I'm thinking of is calculating the mean on axis=1 (mean embedding of tokens in the sentence) since axis=0 and axis=2 have always the same size :

cos = torch.nn.CosineSimilarity(dim=1) #dim becomes 1 now

u = u.mean(axis=1)
v = v.mean(axis=1)

cos(u, v).detach().numpy().item() # works now and gives 0.7269

But, I'm afraid that I'm hurting the embedding when calculating the mean!

=> The second solution is to pad shorter sentences out, that means:

• giving a list of sentences to embed at a time (instead of embedding sentence by sentence)
• look up for the sentence with the longest tokens and embed it, get its shape S
• for the rest of sentences embed then pad zero to get the same shape S (the sentence has 0 in the rest of dimensions)

What are your thoughts? What technique would you use and why?

Answer 1

I don't think there will be a definitive answer, but I suspect that you'll get better results using the averaging method rather than the padding method.

One big problem with the padding method is that it's sensitive to word order. For example, the sentences "Gibbons are one type of ape" and "One type of ape is the Gibbon" look very different if we do a word-by-word comparison. "Gibbons" is very different than "One", "are" is very different than "type", etc. This problem will be offset slightly because RoBERTa embeddings are context-sensitive, but you get the idea.

Another big problem is that the padding can dominate the similarity comparison for sentences with different lengths. Suppose we want to compare the following two sentences which have very similar meaning:

Sentence1: "Very furry" would be an apt description for most mammalian species.

Sentence2: Mammals have hair.

We would add 8 padding tokens to sentence2 in order to give it the same length as sentence1. But then the similarity computation is dominated by the padding tokens and not by the actual content of the sentence.

To me, the averaging approach seems superior because it avoids these two problems. Of course it also has drawbacks. The "sentence embeddings" obtained by averaging the word embeddings will be noisy because each word is given equal weight.

You could try running a keyword detector over the sentences you wish to compare. Then compare the means of the top-k keywords instead of the whole sentence.