New answers tagged transformer
1
vote
Accepted
How many parameters does the vanilla Transformer have?
Table 3 has all the values of the hyper-parameters of the models.
See the image below, green are for the base and blue for the big model.
You can use these to get the matrices sizes. For example for ...
- 3,448
0
votes
Does high number of output labels affect the performance of BERT and how to handle the class imbalance issue while doing multi text classification?
Why does your distribution contains 14 classes? What about the 102 others?
Quick but generic answers:
The number of classes always affect performance, because it's always easier to predict the ...
- 24.5k
0
votes
Using BERT to extract a list of words and phrases from documents
It depends on the context imho: where does the list of words/phrases come from? Did some expert compile it thinking about every word carefully, or is it just intended as some rough indication of the ...
- 24.5k
0
votes
VQ-GAN understanding
GANs are a type of deep learning model that consist of two parts: a generator network that produces fake images, and a discriminator network that tries to distinguish the fake images from real ones. ...
- 2,946
0
votes
Is there bias in matrix multiplications for self attention
Don't know if you got an answer, but I recently had a similar question when looking at a transformer implemented in code.The QKV projection step does not seem to involve a bias, as shown in this code ...
- 1
0
votes
Should I open abbreviations/acronyms in the text data, when training transformer model?
It depends on what you want the model to learn and the size of the dataset.
If it is a generative model and you want the model to generative abbreviations/acronyms, then those have to be in the data.
...
- 19.4k
1
vote
Should I open abbreviations/acronyms in the text data, when training transformer model?
In general, it is usual not to do any preprocessing of the text. This, however, depends heavily on your specific case. I would suggest not doing any preprocessing but evaluating the results on the ...
- 19k
1
vote
Accepted
What does the output of an encoder in encoder-decoder model represent?
Encoder-decoder with RNNs
With RNNs, you can either use the hidden state of the encoder's last time step (i.e. return_sequences=False in Keras) or use the outputs/...
- 19k
1
vote
What does the output of an encoder in encoder-decoder model represent?
If you're using an RNN architecture as your encoder, say an LSTM or GRU, then the output of your encoder is the hidden-state representation of each time step in your input. So for each example that ...
- 31
1
vote
Accepted
Self-attention in Transformers, are the component values of input vector trained or is it the set W_q, W_k, W_v?
The training of a self-attention layer will result in the update of the $W$ matrices and the gradient being propagated back to the previous layer.
At the end of the self-attention blocks, the back-...
- 19k
0
votes
Accepted
What are the advantages of autoregressive over seq2seq?
Encoder-decoder architectures are normally used when there is an input sequence and an output sequence, and the output sequence is generated autoregressively. The encoder processes the whole input ...
- 19k
0
votes
Save and Load Simple Transformer Model
you just need to save each section to a target path
...
0
votes
Is the Transformer decoder an autoregressive model?
The vanilla transformer decoder models proposed in the original paper "Attention Is All You Need" and the OpenAI GPT-series models are autoregressive models at inference time. In addition, I ...
- 39
Top 50 recent answers are included