# Text similarity with sentence embeddings

I'm trying to calculate similarity between texts with various lengths. My current approach is following:

1. Using Universal Sentence Encoder, I convert text to a set of vectors.
2. I average these vectors to create the final feature vector.
3. I compare feature vectors using cosine similarity.

This gives me pretty good results for texts with roughly same sizes, but I was wondering if there is a better approach for the step #2 if texts have different lengths.

• Still should be fine. Cosine similarity is independent of sizes. Those vectors could have any sizes, but the similarity function measures angle between those two vectors. – aminrd Sep 19 '19 at 22:01
• @aminrd yes, but don't you think that averaging a large corpus "dilutes" the feature vector more? It won't be carrying as much semantic information, as the feature vector created based off a shorter corpus. – Kertis van Kertis Sep 20 '19 at 12:17
• Can you provide us data that represent your text? Do you only want method with cosine similarity or any text similarity method would work? – Mohit Motwani Sep 23 '19 at 13:51
• @MohitMotwani It's not domain-specific texts, so the content shouldn't matter. Any text similarity method would work, as long as it uses results of Universal Sentence Encoder from the step 1. – Kertis van Kertis Sep 23 '19 at 15:21
• I quickly looked up the U.S.E. but it seems to be an encoder based on embeddings. Which means that you'll get a representation that expresses the semantics. I would imagine that most of the stylistic features will be gone, which might be ok if that is what you want. You could introduce new elements to encode those (such as vector length). – S van Balen Sep 23 '19 at 15:57

## 2 Answers

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.

The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

WMD can be modified to Sentence Mover’s Distance, comparing how far apart different sentence embeddings are to each other.

• How's this applicable to Universal Sentence Encoder? My embeddings are at sentence level, not at word level. – Kertis van Kertis Sep 23 '19 at 20:06
• From my understanding of your question, you are embedding sentences and then averaging sentences vectors from different texts. An alternative is doing a variation of WMD - Sentence Mover’s Distance, comparing how far apart different sentence embeddings are homes.cs.washington.edu/~nasmith/papers/… – Brian Spiering Sep 24 '19 at 14:35
• Could you please include this comment in the answer, I would like to mark it as accepted – Kertis van Kertis Sep 27 '19 at 15:54

With the advancement in language models, representation of sentences into vectors has been getting better lately. That might give some good result in your case. For example, BERT can be used to get the sentence embedding. Look at the following usage of BERT for sentence similarity :

You can use the pre-trained BERT model and you can pass two sentences and you can let the vector obtained at C pass through a feed forward neural network to decide whether the sentences are similar. This approach can work if you have labelled set of data. If you don't have, consider the following :

You pass the variable length sentences to the BERT network and the vector obtained at the token C becomes the vector for the sentence. You can then use cosine similarity the way you have been using.

• Please read my question, I'm not asking about sentence level similarity. – Kertis van Kertis Sep 23 '19 at 20:03
• What I suggested will actually give you an end to end way of doing what you want. Averaging sentence vectors to get final vector for the final text is a common way but has drawbacks like you said. if you use the approach i told you won't be needing to aggregate sentence vectors. You can directly get the vector for your text. – Gyan Ranjan Sep 24 '19 at 10:03
• From what I read, you can't get a document vector directly from BERT, since it has a limit on the sequence length. – Kertis van Kertis Sep 24 '19 at 15:13
• Not only it has a hard limit for the length of the sentence, you either have to average all the word embeddings (not as strong as USE) or you need fine-tune it to get meaningful results from [CLS]: github.com/google-research/bert/issues/164 – Maziyar Dec 20 '19 at 21:17