I'm loading a language model from torch hub (CamemBERT a French RoBERTa-based model) and using it do embed some sentences:
import torch camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0') camembert.eval() # disable dropout (or leave in train mode to finetune) def embed(sentence): tokens = camembert.encode(sentence) # Extract all layer's features (layer 0 is the embedding layer) all_layers = camembert.extract_features(tokens, return_all_hiddens=True) embeddings = all_layers return embeddings # Here we see that the shape of the embedding vector depends on the number of tokens in the sentence u = embed("Bonjour, ça va ?") u.shape # torch.Size([1, 7, 768]) v = embed("Salut, comment vas-tu ?") v.shape # torch.Size([1, 9, 768])
Imagine now, I want to calculate the
cosine distance between the vectors (tensors in our case)
cos = torch.nn.CosineSimilarity(dim=0) cos(u, v) #will throw an error since the shape of `u` is different from the shape of `v``
I'm asking what is the best method to use in order to always get the same embedding shape for a sentence regardless the count of tokens?
=> The first solution I'm thinking of is calculating the
mean on axis=1 (mean embedding of tokens in the sentence) since axis=0 and axis=2 have always the same size :
cos = torch.nn.CosineSimilarity(dim=1) #dim becomes 1 now u = u.mean(axis=1) v = v.mean(axis=1) cos(u, v).detach().numpy().item() # works now and gives 0.7269
But, I'm afraid that I'm hurting the embedding when calculating the mean!
=> The second solution is to pad shorter sentences out, that means:
- giving a list of sentences to embed at a time (instead of embedding sentence by sentence)
- look up for the sentence with the longest tokens and embed it, get its shape
- for the rest of sentences embed then pad zero to get the same shape
S(the sentence has 0 in the rest of dimensions)
What are your thoughts? What technique would you use and why?