4
In order to make your model more robust to different wordings, you may try with data augmentation techniques, that is, creating variations of your sentences and adding them to the training set with the same label as the original sentence.
There are frameworks like TextAttack that offer several text augmentation techniques. Another option is using back-...
4
After a Googling around, I think this tutorial may suit your needs.
However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-based decoders are autoregressive by nature, Transformers are not. Instead, all predictions are generated at once based on the real target tokens (i.e. teacher ...
3
It appears that your model is failing to generalize.
One option is to increase the amount and quality of the training data.
Other options include large-scale language model specific regularization such as mixout and AUBER.
1
For your first question, you can check if the tokenizer covers a certain string with the following:
text = 'today is a good day 😃'
ids2string = lambda ids: tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(ids))
ids2string(tokenizer(text)['input_ids'])
> <s>today is a good day 😃</s>
If emoji is not included in the tokenizer ...
1
T5 is in fact a sequence-to-sequence model, it has an encoder that generates some hidden states representing the input and a decoder that generates the output. When you fine-tune the model you can happily ignore how the model was pre-trained and only train for your specific task as schematically shown in the original Google blog post.
For fine-tuning, you ...
1
Metrics for Q&A
F1 score: Captures the precision and recall that words chosen as being part of the answer are actually part of the answer
EM Score(exact match): which is the number of answers that are exactly correct (with the same start and end index). EM is 1 when characters of model prediction exactly matches True answers.
The above scores are ...
1
EM (exact match) and F1 scores are typically calculated on different levels. EM is calculated on the character level. F1 is calculated on individual word level.
Almost always, EM will be lower than F1. There is a good chance something is incorrect in the code.
You should confirm your assumption by calculating the EM and F1 scores separately for empty answers ...
1
First I suggest reading the transformers paper. Couple of quick notes is that this model consists of an encoder and a decoder, and the original task the paper is trained on is machine translation. Datasets (benchmarks) they used to train and evaluate this model from scratch were WMT 2014 Engligh-to-German, WMT 2014 English-to-French (section 5.1 of the paper)...
Only top voted, non community-wiki answers of a minimum length are eligible