6
The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not possible for the model to index the positional embedding for positions greater than the maximum.
This limitation, nevertheless, is not arbitrary, but has a deeper ...
3
After a Googling around, I think this tutorial may suit your needs.
However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-based decoders are autoregressive by nature, Transformers are not. Instead, all predictions are generated at once based on the real target tokens (i.e. teacher ...
1
T5 is in fact a sequence-to-sequence model, it has an encoder that generates some hidden states representing the input and a decoder that generates the output. When you fine-tune the model you can happily ignore how the model was pre-trained and only train for your specific task as schematically shown in the original Google blog post.
For fine-tuning, you ...
1
Metrics for Q&A
F1 score: Captures the precision and recall that words chosen as being part of the answer are actually part of the answer
EM Score(exact match): which is the number of answers that are exactly correct (with the same start and end index). EM is 1 when characters of model prediction exactly matches True answers.
The above scores are ...
1
EM (exact match) and F1 scores are typically calculated on different levels. EM is calculated on the character level. F1 is calculated on individual word level.
Almost always, EM will be lower than F1. There is a good chance something is incorrect in the code.
You should confirm your assumption by calculating the EM and F1 scores separately for empty answers ...
1
First I suggest reading the transformers paper. Couple of quick notes is that this model consists of an encoder and a decoder, and the original task the paper is trained on is machine translation. Datasets (benchmarks) they used to train and evaluate this model from scratch were WMT 2014 Engligh-to-German, WMT 2014 English-to-French (section 5.1 of the paper)...
1
For your first question, you can check if the tokenizer covers a certain string with the following:
text = 'today is a good day 😃'
ids2string = lambda ids: tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(ids))
ids2string(tokenizer(text)['input_ids'])
> <s>today is a good day 😃</s>
If emoji is not included in the tokenizer ...
1
A recent PR changed the location of the scripts you are looking for to examples/legacy/question-answering
1
For Question Answering, you need 2 logits : one for the start position, one for the end position. Based on these 2 logits, you have an answer span (denoted by the start/end position).
In the source code, you have : pooled_output = self.pooler(sequence_output)
If you take a look at the pooler, there is a comment :
# We "pool" the model by simply taking the ...
Only top voted, non community-wiki answers of a minimum length are eligible