I am working with question answering and machine reading comprehension system. I want to match questions and documents (around 100,000 docs) in database. I've used tf-idf but it accuracy is about 55% and I need to be at least 80%. Could you give me some advice?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
I am still not quite sure how you are solving in the problem with tf-idf for a QA system. However, there have been lots of improvements that has been done in the QA domain over the years, with the usage of deep learning for natural language processing.
I would urge you to look at the following approaches that might help you to reach the accuracies you are looking for:
- Bidaf Model that was used to solve SQuAD dataset.
- BERT model that was used to solve the SQuAD dataset.
You have multiple open source implementations to train the network and use it for prediction. You could also learn about them in the papers published by the creators of these networks.
