New answers tagged

1 vote
Accepted

How many parameters does the vanilla Transformer have?

Table 3 has all the values of the hyper-parameters of the models. See the image below, green are for the base and blue for the big model. You can use these to get the matrices sizes. For example for ...
0 votes

Does high number of output labels affect the performance of BERT and how to handle the class imbalance issue while doing multi text classification?

Why does your distribution contains 14 classes? What about the 102 others? Quick but generic answers: The number of classes always affect performance, because it's always easier to predict the ...
  • 24.5k
0 votes

Using BERT to extract a list of words and phrases from documents

It depends on the context imho: where does the list of words/phrases come from? Did some expert compile it thinking about every word carefully, or is it just intended as some rough indication of the ...
  • 24.5k
0 votes

VQ-GAN understanding

GANs are a type of deep learning model that consist of two parts: a generator network that produces fake images, and a discriminator network that tries to distinguish the fake images from real ones. ...
  • 2,946
0 votes

Is there bias in matrix multiplications for self attention

Don't know if you got an answer, but I recently had a similar question when looking at a transformer implemented in code.The QKV projection step does not seem to involve a bias, as shown in this code ...
  • 1
0 votes

Should I open abbreviations/acronyms in the text data, when training transformer model?

It depends on what you want the model to learn and the size of the dataset. If it is a generative model and you want the model to generative abbreviations/acronyms, then those have to be in the data. ...
1 vote

Should I open abbreviations/acronyms in the text data, when training transformer model?

In general, it is usual not to do any preprocessing of the text. This, however, depends heavily on your specific case. I would suggest not doing any preprocessing but evaluating the results on the ...
  • 19k
1 vote
Accepted

What does the output of an encoder in encoder-decoder model represent?

Encoder-decoder with RNNs With RNNs, you can either use the hidden state of the encoder's last time step (i.e. return_sequences=False in Keras) or use the outputs/...
  • 19k
1 vote

What does the output of an encoder in encoder-decoder model represent?

If you're using an RNN architecture as your encoder, say an LSTM or GRU, then the output of your encoder is the hidden-state representation of each time step in your input. So for each example that ...
  • 31
1 vote
Accepted

Self-attention in Transformers, are the component values of input vector trained or is it the set W_q, W_k, W_v?

The training of a self-attention layer will result in the update of the $W$ matrices and the gradient being propagated back to the previous layer. At the end of the self-attention blocks, the back-...
  • 19k
0 votes
Accepted

What are the advantages of autoregressive over seq2seq?

Encoder-decoder architectures are normally used when there is an input sequence and an output sequence, and the output sequence is generated autoregressively. The encoder processes the whole input ...
  • 19k
0 votes

Save and Load Simple Transformer Model

you just need to save each section to a target path ...
0 votes

Is the Transformer decoder an autoregressive model?

The vanilla transformer decoder models proposed in the original paper "Attention Is All You Need" and the OpenAI GPT-series models are autoregressive models at inference time. In addition, I ...
  • 39

Top 50 recent answers are included