1
$\begingroup$

According to BERT author Jacob Devlin: I'm not sure what these vectors are, since BERT does not generate meaningful sentence vectors. It seems that this is doing average pooling over the word tokens to get a sentence vector, but we never suggested that this will generate meaningful sentence representations. And even if they are decent representations when fed into a DNN trained for a downstream task, it doesn't mean that they will be meaningful in terms of cosine distance. (Since cosine distance is a linear space where all dimensions are weighted equally).

$\endgroup$
1
$\begingroup$

They might or might not be similar, the embeddings extracted by mean pooling the BERT output usually have high cosine similarity even though the input sentences are completely different.

Bert embeddings are not meant for sentence similarity task(SST), but there is some research combining Bert and SST. Here are those resources,

SBERT paper: https://arxiv.org/abs/1908.10084

SBERT implementation: https://github.com/UKPLab/sentence-transformers

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.