New answers tagged transformer
0
I'm unclear whether transformers are the best tool for time series forecasting. Transformers at the end of the day are just the latest in a series of sequence-to-sequence models with an encoder and decoder. This means that transformers change something to something else.
With time series you aren't changing something to something else, you're trying to find ...
0
Although, the previous answer is a good reference to find how to measure probability of a sentence using BERT, in order to perform a meaningful evaluation of cross-model (e.g., compare BERT with Roberta) they should use the same tokenization.
3
After a Googling around, I think this tutorial may suit your needs.
However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-based decoders are autoregressive by nature, Transformers are not. Instead, all predictions are generated at once based on the real target tokens (i.e. teacher ...
2
We find some justifications in the Conformer paper:
Convolutions are better than Transformers at detecting fine-grained patterns:
While Transformers are good at modeling long-range global context, they are less capable to extract fine-grained local feature patterns. Convolution neural networks (CNNs), on the other hand, exploit local information and are
...
1
Self-attention means X pays attention to X, as opposed to "normal" attention where X pays attention to Y.
Multi-head attention is as opposed to single-head attention. You can choose to use multi- or single-head attention equally for self-attention and for normal-attention.
Masking X and/or Y is a third independent aspect of a design.
In a ...
0
I like to think about it in the context of progression of attention mechanisms in neural networks.
Early attention mechanisms were implemented as explicit sliding windows over the encoding sequence. For example, Graves 2013 implemented it as the average of several sliding Gaussians. So, it was a local attention mechanism to attend to a learnable window ...
0
A common intuition is viewing attention as probabilistic retrieval: given a query, you want to retrieve some information from some states (values) given some keys that describe the values.
With single-head attention, you get a weighted average of some hidden states with everything that is contained in there. Once you introduce, the head-specific projection, ...
0
In this context, masking means replacing the token with a special [MASK] token. The network does not have the information of what the original token was, the only way how it could potentially figure out what it was is by looking at the context.
It is not the loss function that guarantees that the model learns something meaningful, it is architecture design ...
1
the first few bits of the embedding are completely unusable by the
network because the position encoding will distort them a lot
This confused me very much at first because I was thinking of the model using a pre-trained word embedding. And then an arbitrary initial chunk of that embedding gets severely tampered with by the positional encoding.
However, in ...
0
It is the encoder part of the Transformer model that is bidirectional in nature, not the whole model.
The full Transformer model has two parts: encoder and decoder. This encoder-decoder model is used for sequence-to-sequence tasks, like machine translation.
There are other tasks, however, that do not need the full model, but only one of its parts. For ...
0
(This answer was originally a comment)
You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very ...
Top 50 recent answers are included
