8
$\begingroup$

I'm trying to calculate similarity between texts with various lengths. My current approach is following:

  1. Using Universal Sentence Encoder, I convert text to a set of vectors.
  2. I average these vectors to create the final feature vector.
  3. I compare feature vectors using cosine similarity.

This gives me pretty good results for texts with roughly same sizes, but I was wondering if there is a better approach for the step #2 if texts have different lengths.

Improve this question
$\endgroup$
6
  • $\begingroup$ Still should be fine. Cosine similarity is independent of sizes. Those vectors could have any sizes, but the similarity function measures angle between those two vectors. $\endgroup$
    – aminrd
    Sep 19, 2019 at 22:01
  • $\begingroup$ @aminrd yes, but don't you think that averaging a large corpus "dilutes" the feature vector more? It won't be carrying as much semantic information, as the feature vector created based off a shorter corpus. $\endgroup$
    – Kertis van Kertis
    Sep 20, 2019 at 12:17
  • $\begingroup$ Can you provide us data that represent your text? Do you only want method with cosine similarity or any text similarity method would work? $\endgroup$
    – Mohit Motwani
    Sep 23, 2019 at 13:51
  • $\begingroup$ @MohitMotwani It's not domain-specific texts, so the content shouldn't matter. Any text similarity method would work, as long as it uses results of Universal Sentence Encoder from the step 1. $\endgroup$
    – Kertis van Kertis
    Sep 23, 2019 at 15:21
  • $\begingroup$ I quickly looked up the U.S.E. but it seems to be an encoder based on embeddings. Which means that you'll get a representation that expresses the semantics. I would imagine that most of the stylistic features will be gone, which might be ok if that is what you want. You could introduce new elements to encode those (such as vector length). $\endgroup$
    – S van Balen
    Sep 23, 2019 at 15:57

2 Answers 2

Reset to default
8
+50
$\begingroup$

One approach is using Word Mover’s Distance (WMD). WMD is an algorithm for finding the distance between texts of different lengths, where each word is represented as a word embedding vector.

The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to "travel" to reach the embedded words of another document.

For example:

enter image description here Source: "From Word Embeddings To Document Distances" Paper

WMD can be modified to Sentence Mover’s Distance, comparing how far apart different sentence embeddings are to each other.

Improve this answer
$\endgroup$
3
  • $\begingroup$ How's this applicable to Universal Sentence Encoder? My embeddings are at sentence level, not at word level. $\endgroup$
    – Kertis van Kertis
    Sep 23, 2019 at 20:06
  • $\begingroup$ From my understanding of your question, you are embedding sentences and then averaging sentences vectors from different texts. An alternative is doing a variation of WMD - Sentence Mover’s Distance, comparing how far apart different sentence embeddings are homes.cs.washington.edu/~nasmith/papers/… $\endgroup$
    – Brian Spiering
    Sep 24, 2019 at 14:35
  • $\begingroup$ Could you please include this comment in the answer, I would like to mark it as accepted $\endgroup$
    – Kertis van Kertis
    Sep 27, 2019 at 15:54
0
$\begingroup$

With the advancement in language models, representation of sentences into vectors has been getting better lately. That might give some good result in your case. For example, BERT can be used to get the sentence embedding. Look at the following usage of BERT for sentence similarity : BERT for sentence similarity

You can use the pre-trained BERT model and you can pass two sentences and you can let the vector obtained at C pass through a feed forward neural network to decide whether the sentences are similar. This approach can work if you have labelled set of data. If you don't have, consider the following :

BERT for single sentence

You pass the variable length sentences to the BERT network and the vector obtained at the token C becomes the vector for the sentence. You can then use cosine similarity the way you have been using.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Please read my question, I'm not asking about sentence level similarity. $\endgroup$
    – Kertis van Kertis
    Sep 23, 2019 at 20:03
  • 1
    $\begingroup$ What I suggested will actually give you an end to end way of doing what you want. Averaging sentence vectors to get final vector for the final text is a common way but has drawbacks like you said. if you use the approach i told you won't be needing to aggregate sentence vectors. You can directly get the vector for your text. $\endgroup$
    – Gyan Ranjan
    Sep 24, 2019 at 10:03
  • $\begingroup$ From what I read, you can't get a document vector directly from BERT, since it has a limit on the sequence length. $\endgroup$
    – Kertis van Kertis
    Sep 24, 2019 at 15:13
  • $\begingroup$ Not only it has a hard limit for the length of the sentence, you either have to average all the word embeddings (not as strong as USE) or you need fine-tune it to get meaningful results from [CLS]: github.com/google-research/bert/issues/164 $\endgroup$
    – Maziyar
    Dec 20, 2019 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.