According to BERT author Jacob Devlin: I'm not sure what these vectors are, since BERT does not generate meaningful sentence vectors. It seems that this is doing average pooling over the word tokens to get a sentence vector, but we never suggested that this will generate meaningful sentence representations. And even if they are decent representations when fed into a DNN trained for a downstream task, it doesn't mean that they will be meaningful in terms of cosine distance. (Since cosine distance is a linear space where all dimensions are weighted equally).
1
-
$\begingroup$ For the benefit of other users in future: For a more detailed discussion on this topic, see this Stackoverflow Q&A - stackoverflow.com/questions/60492839/… $\endgroup$– so2Commented Jul 6 at 19:07
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
They might or might not be similar, the embeddings extracted by mean pooling the BERT output usually have high cosine similarity even though the input sentences are completely different.
Bert embeddings are not meant for sentence similarity task(SST), but there is some research combining Bert and SST. Here are those resources,
SBERT paper: https://arxiv.org/abs/1908.10084
SBERT implementation: https://github.com/UKPLab/sentence-transformers
