New answers tagged transformer
1
At training time, the input to the decoder is the target sentence tokens, which are indeed unknown at the test time. What you call the second input are the desired outputs, which are not usually referred to as an input to the decoder, 1. for clarity, 2. they are technically input to the loss function.
At test time, we do not need the loss function, but we ...
6
The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not possible for the model to index the positional embedding for positions greater than the maximum.
This limitation, nevertheless, is not arbitrary, but has a deeper ...
0
So, the question talks about how to treat transformation choices as hyper parameters.
How I would go about it is the following:
Use one baseline model architecture for the data and then repeat the following:
Instantiate the baseline model (effectively make sure all of the weights are initialised)
Create the transformed dataset
Train the model
Compute ...
0
What makes you think your model is overfitting? Are you concerned about the difference between the training loss and validation loss?
If so, this is not overfitting. Overfitting is when the weights learned from training fail to generalize to data unseen during model training.
In the case of the plot shown here, your validation loss continues to go down, so ...
1
The trick is that you do not need masking at inference time. The purpose of masking is that you prevent the decoder state from attending to positions that correspond to tokens "in the future", i.e., those that will not be known at the inference time, because they will not have been generated yet.
At inference time, it is no longer a problem because ...
1
Both approaches are reasonable. Updating the BERT weights will train for longer period of time, but should give more accurate results.
Top 50 recent answers are included
