4
$\begingroup$

I am looking for the correct model / approach for the task of checking if two sentences have the same meaning

I know I can use embeddings to check similarity, but that is not what I am after. I suspect BERT style LLM have nice higher level vector that mights be useful, but I'm not sure how to apply that.

For example this sentence:

  • I am very lazy

Has a somewhat similar meaning as:

  • I don't like to work hard

But not

  • A lazy horse is not very useful

Using 'just' embeddings (for example HF: allMiniLM-L6-v2) gives results that are not useful.

enter image description here

What would be a good appoarch?

$\endgroup$

1 Answer 1

4
$\begingroup$

The similarity used to train this model might be different from the similarity you expect.

A better approach would be create your own large and good quality training set of similar and dissimilar sentences and fine-tune a pretrained model (the one from your question or some other) using the same sentence transformers library (https://www.sbert.net).

Another currently available alternative is to play with prompts for the huge commercial models (ChatGPT, GPT-4, Google Bard, etc) and hopefully they can understand what you want and do the task for you without any additional effort. For example ChatGPT said sentence B is more similar to A in 10/10 retries in my test.

$\endgroup$
3
  • 2
    $\begingroup$ As I understood LLM foundation models like BERT, have the more 'generalized' meaning of the underlying sentence in the higher level layers; so I'm looking to exploit that so I can do this task locally instead of having to resort to huge external models. Maybe a sentence-transformer is not the ideal match? $\endgroup$ Dec 8, 2023 at 12:18
  • 3
    $\begingroup$ sentence-transformers is a great method. I think the problem in your case is the particular model you chose (allMiniLM-L6-v2). Look at the description of how it was fine-tuned: "given a sentence from the pair, the model should predict which out of a set of randomly sampled other sentences, was actually paired with it in our dataset". So for this model two sentences are "similar" if they are likely to be next to each other. This explains why this model prefers C to B. For "meaning" you need to further fine-tune it. $\endgroup$
    – Valentas
    Dec 8, 2023 at 14:42
  • $\begingroup$ Ah I see, thanks for the clarification. In the meanwhile I read more about the sentence-transformers architecture and it indeed seems a good match. Thanks! $\endgroup$ Dec 8, 2023 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.