# Tag Info

1 vote
Accepted

### How many parameters does the vanilla Transformer have?

Table 3 has all the values of the hyper-parameters of the models. See the image below, green are for the base and blue for the big model. You can use these to get the matrices sizes. For example for ...
• 3,448

### Does high number of output labels affect the performance of BERT and how to handle the class imbalance issue while doing multi text classification?

Why does your distribution contains 14 classes? What about the 102 others? Quick but generic answers: The number of classes always affect performance, because it's always easier to predict the ...
• 24.5k

### Using BERT to extract a list of words and phrases from documents

It depends on the context imho: where does the list of words/phrases come from? Did some expert compile it thinking about every word carefully, or is it just intended as some rough indication of the ...
• 24.5k

### VQ-GAN understanding

GANs are a type of deep learning model that consist of two parts: a generator network that produces fake images, and a discriminator network that tries to distinguish the fake images from real ones. ...
• 2,946

### Is there bias in matrix multiplications for self attention

Don't know if you got an answer, but I recently had a similar question when looking at a transformer implemented in code.The QKV projection step does not seem to involve a bias, as shown in this code ...

### Should I open abbreviations/acronyms in the text data, when training transformer model?

It depends on what you want the model to learn and the size of the dataset. If it is a generative model and you want the model to generative abbreviations/acronyms, then those have to be in the data. ...
• 19.4k
1 vote

### Should I open abbreviations/acronyms in the text data, when training transformer model?

In general, it is usual not to do any preprocessing of the text. This, however, depends heavily on your specific case. I would suggest not doing any preprocessing but evaluating the results on the ...
• 19k
1 vote
Accepted

### What does the output of an encoder in encoder-decoder model represent?

Encoder-decoder with RNNs With RNNs, you can either use the hidden state of the encoder's last time step (i.e. return_sequences=False in Keras) or use the outputs/...
• 19k
1 vote

### What does the output of an encoder in encoder-decoder model represent?

If you're using an RNN architecture as your encoder, say an LSTM or GRU, then the output of your encoder is the hidden-state representation of each time step in your input. So for each example that ...
• 31
1 vote
Accepted

### Self-attention in Transformers, are the component values of input vector trained or is it the set W_q, W_k, W_v?

The training of a self-attention layer will result in the update of the $W$ matrices and the gradient being propagated back to the previous layer. At the end of the self-attention blocks, the back-...
• 19k
Accepted

### What are the advantages of autoregressive over seq2seq?

Encoder-decoder architectures are normally used when there is an input sequence and an output sequence, and the output sequence is generated autoregressively. The encoder processes the whole input ...
• 19k