We are using Google BERT for Question and Answering. We have fine tuned BERT with SQUAD QnA release train data set (https://github.com/google-research/bert , https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)
It generated new checkpoints and BERT is giving good answers for most of questions we asked on our text documents. However, there are some questions which it is answering wrong, so we are trying to further fine tune with our Question and known answer on our text document. We further trained based on last generated checkpoint and got new checkpoint.
With new checkpoint when we are asking the same question, the answer did not got corrected! Previously BERT was giving wrong answer with 99% confidence and now also giving same wrong answer with 95% confidence.
Can someone suggest, if they have same or similar experience, and suggest please.
Following are questions in BERT github Issues, and are unanswered for quite some time:
- BERT accuracy reduced after providing custom training..The answer is also not correct : https://github.com/google-research/bert/issues/492
- Unable to incrementally train BERT with custom training: https://github.com/google-research/bert/issues/482
- Little training has no impact: https://github.com/google-research/bert/issues/481
- Custom Domain Training: https://github.com/google-research/bert/issues/498