2
$\begingroup$

We are using Google BERT for Question and Answering. We have fine tuned BERT with SQUAD QnA release train data set (https://github.com/google-research/bert , https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)

It generated new checkpoints and BERT is giving good answers for most of questions we asked on our text documents. However, there are some questions which it is answering wrong, so we are trying to further fine tune with our Question and known answer on our text document. We further trained based on last generated checkpoint and got new checkpoint.

With new checkpoint when we are asking the same question, the answer did not got corrected! Previously BERT was giving wrong answer with 99% confidence and now also giving same wrong answer with 95% confidence.

Can someone suggest, if they have same or similar experience, and suggest please.
Following are questions in BERT github Issues, and are unanswered for quite some time:

$\endgroup$
2
$\begingroup$

Remember that BERT was first pre-trained using the concatenation of BooksCorpus (800M words) and English Wikipedia (2,500M words). Then the fine-tuning in your case uses the SQuAD dataset consisting of 100,000+ questions (based on Wikipedia articles) with a learning rate in the order of e-5.

So when you add let's say 100 new domain-specific question/document/answer to your input during the fine-tuning, this will not have a major effect on the learned parameters especially if your new input has out-of-Bert-vocabulary words or uses different English style, as opposed to Wikipedia/Books style.

Can someone suggest, if they have same or similar experience, and suggest please.

Following are some suggestions that might help:

  • Increase the amount of new input data to help the model converge to your domain-specific use case.

  • Before fine-tuning, map recurrent out-of-Bert-vocabulary words of your new input to existing words in Bert vocabulary which have almost the same meaning.

  • Increase the learning rate for your new input. But this is not very safe as the model may "forget" what he learned and not converge. But as the fine-tuning doesn't take much time you can try it out and see the results.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for reply. We have tried step 1, means gave question and answer multiple times, but it did not have any impact on the output. $\endgroup$ – Sandeep Bhutani Apr 23 '19 at 13:17
  • $\begingroup$ I suggest giving new question/answer and not repeating the same ones many times. Also try to combine with other suggested solutions as they all can be mixed together. I believe with SQuAD and hundreds of new question/answer you'll get the results. But if your vocabulary is so different it's almost impossible without mapping (step2) if applicable or pre-training from scratch (needs millions of input data and computational power). $\endgroup$ – HLeb Apr 23 '19 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.