1
$\begingroup$

I am having a question answering system which is using Seq2Seq kind of architecture. Actually it is a transformer architecture. When a question is asked it gives startposition and endposition of answer along with their logits.
The answer is formed by choosing the best logits span and final probability is calculated by summing the start and end logits.

Now the problem is, I have multiple answer and many times the good answer is at 2nd or 3rd place (after sorting on the result of sum of start and end probability). Is there any metric in search engine science using which I can rank the best answers?

Followings have been tried:

  • cosine similarity between question words and answers - This works many times but fails when question semantic meaning is complex
  • TFIDF - gives good score but fails when there is synonym in answers rather than matching word.
  • gensim semantic similarity - fails badly.
  • BLUE score and new BERTF1Score also tried

Few terms I heard of but I doubt if these work, like Mean Reciprocal Rank which I think gives search quality rather than answer quality and also the correct response is required to calculate MRR (Please correct if I am wrong). Or the PageRank which is not valid in my case as the answer semantic meaning is preferred in QnA rather than the document popularity.

Kindly suggest other metrics which search engines generally use to rank the answers.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The ranking of the answers is part of the ML process, i.e. a system should be trained to rank the answers according to their relevance. Heuristic measures such as the ones mentioned in your question may offer decent approximations, but as you noticed they are very limited.

You may be interested in datasets and methods used in shared tasks about QA, for instance https://mrqa.github.io/shared.

Improve this answer
$\endgroup$
5
  • $\begingroup$ These are datasets. I am looking some metric to qualify the answers generated by model. Did I miss something that you want to say? Would you like to point and elaborate how this training data helps my problem? $\endgroup$
    – Sandeep Bhutani
    Dec 21, 2019 at 19:14
  • $\begingroup$ Are you trying to evaluate the model or to rank the answers as part of the output of the system? In the latter case as far as I know my answer is how it's done in state of the art systems: it's not a heuristic, it's part of the ML process (comparable to recommender systems for instance). $\endgroup$
    – Erwan
    Dec 21, 2019 at 19:22
  • $\begingroup$ Hi.. I am looking to rank the answer $\endgroup$
    – Sandeep Bhutani
    Dec 24, 2019 at 6:48
  • $\begingroup$ @SandeepBhutani ok, my answer means that I think you're mistaken to try to find a heuristic (i.e. an unsupervised deterministic method) which can achieve a good ranking: in theory this heuristic would have to correctly rank any set of candidate answers according to their relevance, so if this was possible you wouldn't even need to use a ML system in the first place. That's why the process of finding the right answer among the set of answers should be part of your ML system, e.g. you could have a second part of the system which has to predict the best answer from the set of candidate answers. $\endgroup$
    – Erwan
    Dec 24, 2019 at 11:30
  • $\begingroup$ Also the point of the link to the shared task is that you can read about the methods used by the authors of the best systems, maybe even find the code for some of these systems. $\endgroup$
    – Erwan
    Dec 24, 2019 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.