Skip to main content
3 votes
Accepted

Cross-attention mask in Transformers

From the paper: "We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This masking, combined with fact that the output ...
ИванКарамазов's user avatar
3 votes

Cross-attention mask in Transformers

I don't understand if we should combine the causal mask with the padding mask from the encoder output or if we should just apply the padding mask (since the VALUES are coming from the encoder, and we ...
Valentin Calomme's user avatar
2 votes
Accepted

Decoder Transformer feedforward

There are some problems with your description: During training, the decoder receives all the shifted target tokens, prepending the BOS token. You removed sole. The ...
noe's user avatar
  • 27k
1 vote

Anonymize continuous variable for masking purposes

I think you can use a much more complicated monotonic transformation, like log(1.234578 + sqrt(x + 7.4142) ** 3) which will be harder to invert than a simple log. ...
David Masip's user avatar
  • 6,101
1 vote

Why shouldn't we mask [CLS] and [SEP] in preparing inputs for a MLM?

In practice, nothing is preventing one from doing what you propose, masking and predicting the [CLS] or the [SEP] token. But the important question is why the model would need to learn about unmasking ...
Tanzir Pial's user avatar
1 vote

Dealing with high frequency tokens during masked Language modelling?

The goal of language modeling is to build a statistical model of how language is used in a specific context. One of the important components of that is token frequency. Bias can mean many things in ...
Brian Spiering's user avatar
1 vote
Accepted

There could be a problem with the linear layer after the attention inside a transformer?

No, this is not a problem. If we zoom into the scaled dot product attention blocks, which happen before the projection with $W^O$ we see this: There, you can see how the masking of the current and ...
noe's user avatar
  • 27k

Only top scored, non community-wiki answers of a minimum length are eligible