I've a sentence 'A' which is a possible response to one of the x number of questions. I need to compare the response with each sentence to see which question's response sentence 'A' is or not a response at all. Any ideas on how to go about this. Pure nlp based solution will be helpful as there are no existing labelled data for the same.
3
-
$\begingroup$ Check out the SQUAD dataset and it's leaderboards. Most of the top scorers are BERT based models. rajpurkar.github.io/SQuAD-explorer if this is overkill there are other solutions such as SVD or simple binary searches (ctrl+F type of search). $\endgroup$– bstrainCommented May 18, 2020 at 18:24
-
$\begingroup$ Can you please elborate on this.? I'm pretty new and I reiterate, I do not have a training set for this. And those trained on others, how well it'll fare against mine? I used a question detector trained on nps_chat(nltk) and it failed miserably. $\endgroup$– AbishakeCommented May 19, 2020 at 13:36
-
$\begingroup$ The SQuAD is one of the relatively few question and answer training sets. NLP is challenging because there isn't always a "right answer." If you want a gentler introduction look at gensim's similarity query tutorial: radimrehurek.com/gensim/auto_examples/core/… Finding two similar questions (what you are doing) is basically a similarity query so this should be applicable to your problem. $\endgroup$– bstrainCommented May 19, 2020 at 14:01
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
So, I gathered that your question is about how to evaluate whether a sentence A is a response to a given question Q or not.
Drawing from knowledge of question-answer systems, you can evaluate whether a sentence responds to question Q by comparing the underlying lambda calculus (essentially translates language meaning into a logical, more computation-looking form - https://www.cl.cam.ac.uk/teaching/1314/NLP/slides6.pdf) which make both the question Q and a response A.
To get the lambda calculus for a question Q or response A, we can use an encoder-decoder model to generate these. In an encoder-decoder network architecture, we simply encode input (e.g. question Q using an RNN/LSTM) information into a "hidden representation", which we then subsequently decode into our desired output (e.g. Lambda Calculus, using an RNN/LSTM). This process is actually called semantic parsing and here are a couple of papers which do this and label the architectures they used to generate these, including:
Maybe this can be an implicit way of comparing the questions with the responses, by then looking at similarities in their lambda calculus.
$\begingroup$
$\endgroup$
About the data, you can use any question answering (QA) dataset to build your training and validation datasets. SQuAD is probably the most popular now, but there are many others (see here). From these datasets, you will be able to create a collection of pairs to question and their correct answers and questions and the wrong answers.
About the model, BERT is prepared to classify pairs of sentences. Originally, it classified whether the second sentence followed the first sentence in the original text (next sentence prediction), but you can fine tune it and repurpose it to receive the question as first sentence and the potential answer as second sentence, and classify whether the answer is actually answering the question.
