All Questions
4 questions
0
votes
1
answer
480
views
Effect of hyperparameters: the hidden size, layers, MLP size number of heads on Transformer
Is there any paper that explains the effect of hyperparameters:
hidden size
Number of layers
MLP size
number of heads
on Transformer performance. I found some explanation on the web but I need ...
- 127
4
votes
1
answer
767
views
Should a Learning Rate Scheduler adjust the learning rate by optimization step (batch) or by epoch?
In PyTorch doc, it suggests
torch.optim.lr_scheduler provides several methods to adjust the learning rate based on the number of epochs.
However, from other ...
- 392
1
vote
0
answers
389
views
Dataset Format for fine tuning deepset/roberta-base-squad2 hugging face transformer model
I have been trying to fine tune the roberta model for QnA to my specific domain (healthcare).
I am unable to find the correct way to provide the dataset format to the tokenizer in order to fine tune ...
- 111
4
votes
1
answer
3k
views
How does the Transformer predict n steps into the future?
I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch ...
- 43