Questions tagged [masking]
The masking tag has no usage guidance.
8
questions
1
vote
0
answers
16
views
Why shouldn't we mask [CLS] and [SEP] in preparing inputs for a MLM?
I know that MLM is trained for predicting the index of MASK token in the vocabulary list, and I also know that [CLS] stands for the beginning of the sentence and [SEP] telling the model the end of the ...
- 11
0
votes
0
answers
14
views
How to produce these tensors efficiently/fast?
I would like to produce the following tensor of size (N*N) where the ones (D) appear as follows:
...
0
votes
0
answers
13
views
BERT performance on the frequency of masking
I was checking out BERT's MLM task, and was wondering if there's any performance impact in terms of accuracy or speed when multiple words are being masked in a sentence.
I have a task where I need to ...
0
votes
0
answers
17
views
Computing Attention on multi-dimensional sequences?
Is it possible to compute attention/adapt existing transformer architectures (like longformer) to be used on multi-dimensional sequence input?
As in, instead of a ...
- 157
1
vote
1
answer
43
views
Dealing with high frequency tokens during masked Language modelling?
Suppose I am working with a Masked Language Model to pre-train on a specific dataset. In that dataset, most sequences have a particular token of a high frequency
...
- 157
0
votes
1
answer
40
views
There could be a problem with the linear layer after the attention inside a transformer?
My question regards this image:
It seems that after the multi head attention there is a linear layer as they mention also from here:
the linearity is given by the weights W^{o}. my quesion is: for ...
- 85
1
vote
1
answer
167
views
Decoder Transformer feedforward
I have a question about the decoder transformer feed forward during training.
Let's pick an example: input data "i love the sun" traduction i want to ...
- 85
1
vote
1
answer
45
views
Anonymize continuous variable for masking purposes
I am about to kick off a large hackathon event.
We have a dataset that is comprised of one continuous variable with high precision, and a number of categorical variables qualifying these data 3-levels ...
- 901