People who code: we want your input. Take the Survey

Questions tagged [masking]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
12 views

Computing Attention on multi-dimensional sequences?

Is it possible to compute attention/adapt existing transformer architectures (like longformer) to be used on multi-dimensional sequence input? As in, instead of a ...
1
vote
0answers
17 views

Dealing with high frequency tokens during masked Language modelling?

Suppose I am working with a Masked Language Model to pre-train on a specific dataset. In that dataset, most sequences have a particular token of a high frequency ...
0
votes
1answer
19 views

There could be a problem with the linear layer after the attention inside a transformer?

My question regards this image: It seems that after the multi head attention there is a linear layer as they mention also from here: the linearity is given by the weights W^{o}. my quesion is: for ...
1
vote
1answer
66 views

Decoder Transformer feedforward

I have a question about the decoder transformer feed forward during training. Let's pick an example: input data "i love the sun" traduction i want to ...
1
vote
1answer
34 views

Anonymize continuous variable for masking purposes

I am about to kick off a large hackathon event. We have a dataset that is comprised of one continuous variable with high precision, and a number of categorical variables qualifying these data 3-levels ...