3
$\begingroup$

I am currently reading Semantic Product Search paper published by Amazon. They are using two evaluation subtasks matching and ranking. In matching, they tune the model hyperparameters to maximize Recall@100 and Mean Average Precision (MAP).

According to Introduction to Information Retrieval, Precision (P) is the fraction of retrieved documents that are relevant:

Recall (R) is the fraction of relevant documents that are retrieved:

I want to know how to come up with ground truth(relevancy label) if it's not available? In other words, if I want to calculate precision or recall for the Semantic product search and if we don't have relevancy label available for input product query. In that case, how researchers calculate precision and recall? or how do they generate it?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I want to know how to come up with ground truth(relevancy label) if it's not available?

There's simply no way to properly evaluate a system if nobody knows what the output is supposed to be. However there are ways to work around a lack of annotated data:

  • Ask a panel of annotators to grade the quality of the output on a sample. Disadvantage: if a relevant instance is never predicted, the annotators are unlikely to notice it.
  • Compare the output to a state of the art system. Disadvantage: the evaluated system can only be as good as the reference system, any error by the reference system is considered correct.
  • Generate artificial data with an automatic method. Disadvantage: the evaluation relies on the quality of the artificial data, so in theory one has to prove that the artificial data is as good as real data... which is usually harder than actually collecting real data.

In that case, how researchers calculate precision and recall? or how do they generate it?

They can't. It would be like grading an exam paper without knowing the correct answers.

This is why benchmark datasets are so important for the research community and are published as proper scientific contributions.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks, Erwan. Can you please point me to some benchmark datasets for the sentence similarity task? $\endgroup$
    – Sayali Sonawane
    Dec 8, 2020 at 10:59
  • 1
    $\begingroup$ @SayaliSonawane sentence similarity is probably too broad, it can mean different things. Paraphrase detection would probably be the most relevant task for semantic similarity between sentence, there are a few datasets used for this task such as this list or this one. $\endgroup$
    – Erwan
    Dec 8, 2020 at 12:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.