Metrics for Q&A
- F1 score: Captures the precision and recall that words chosen as being part of the answer are actually part of the answer
- EM Score(exact match): which is the number of answers that are exactly correct (with the same start and end index). EM is 1 when characters of model prediction exactly matches True answers.
The above scores are computed on individual Q&A pairs. When multiple correct answers are possible for a given question, the maximum score over all possible correct answers is computed. Overall EM and F1 scores are computed for a model by averaging over the individual example scores.
Understanding the basics often solve the questions we are looking for. That being said, you mentioned two things
- Dataset has lot of empty answers
- Same dataset you have used for both training and evaluation [that is the real performance of model is yet to be estimated on a separate dataset]
As no Information of sample dataset nor sample code was provided from your end. Its up to you to find out why EM score is (much) higher than the F1 score by ruling out your assumptions with the elimination rule.
- check with dataset with actual answers but use the same dataset for training and evaluation to confirm the EM score issue is due to dataset having lot of empty answers
- check with separate datasets for training and evaluation, keeping questions with lot of empty answers. Though I quite agree it couldn't be the reason what better way to rule it out by confirming.
Analyse the EM Scores, F1 scores for both scenarios to completely rule out assumptions