3
$\begingroup$

Given a query sentence, we search and find similar sentences in our corpus using transformer-based models for semantic textual similarity.

  • For one query sentence, we might get 200 similar sentences with scores ranging from 0.95 to 0.55.

  • For a second query sentence, we might get 200 similar sentences with scores ranging from 0.44 to 0.27.

  • For a third query sentence, we might only get 100 similar sentences with scores ranging from 0.71 to 0.11.

In all those cases, is there a way to predict where our threshold should be without losing too many relevant sentences? Having a similarity score of 1.0 does not mean that two documents are 2X more similar than if the score was 0.5. Is there a way to determine the topk (how many of the top scoring sentences we should return) parameter?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

As far as I know there is no satisfactory answer:

  • One uses a threshold in order to avoid having to choose a specific K in a top K approach. The threshold is often selected manually to eliminate the sentences which are really not relevant. This makes this method more suitable for favouring recall, if you ask me.
  • Conversely, one uses a "top K" approach in order not to select a threshold. I think K is often selected quite low in order to keep mostly relevant sentences, i.e. it's an approach more suitable for high precision tasks.

The choice depends on the task:

  • First, the approach could be chosen based on what is done with the selected sentences: if it's something like question answering, one wants high precision usually. If it's information retrieval, one wants high recall. If it's a search engine, just rank the sentences by decreasing similarity.
  • Then for the value itself (K or threshold), the ideal case is to do some hyper-parameter tuning. i.e. testing multiple values and evaluate the results. If this is convenient or doable for the task, then look at a few examples and manually select a value which looks reasonable.
Improve this answer
$\endgroup$
2
  • $\begingroup$ If we assume that the task is information retrieval, but it was not domain specific, we can't really tune any hyper parameters, can we? Say, we want to find matches between a specific diabetes drug and medical articles. This is highly specific and will yield low scores. Then we want to find matches between a general philosophical question and articles on philosophy. This is more general and may yield higher scores. I assume we want high recall on both? $\endgroup$
    – DarknessPlusPlus
    Commented Sep 13, 2022 at 12:06
  • $\begingroup$ @DarknessPlusPlus the ideal method would be to: design an evaluation method suited to the task, e.g. have some users annotate how relevant the results are for a set of queries (btw the set of queries should be as representative as possible of the expected queries); then it would be possible to tune the threshold/K using this evaluation method. Of course this takes quite a lot of time and effort so it's not always possible. $\endgroup$
    – Erwan
    Commented Sep 13, 2022 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.