Questions tagged [masking]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
36 views

masking time series data for unsupervised learning

I was studing the A TRANSFORMER-BASED FRAMEWORK FOR MULTIVARIATE TIME SERIES REPRESENTATION LEARNING they frame the unsupervised pre-training as the autoregressive task of denoising the multivariate ...
  • 101
0 votes
0 answers
25 views

Masking pixels for a CNN

I'm trying to implement a CNN with RGB and depth images. But my depth images are a little sparse. So I would like to mask out those neighborhoods in the RGB image where the depth neighbors are empty. ...
1 vote
0 answers
132 views

Why shouldn't we mask [CLS] and [SEP] in preparing inputs for a MLM?

I know that MLM is trained for predicting the index of MASK token in the vocabulary list, and I also know that [CLS] stands for the beginning of the sentence and [SEP] telling the model the end of the ...
  • 11
0 votes
0 answers
15 views

How to produce these tensors efficiently/fast?

I would like to produce the following tensor of size (N*N) where the ones (D) appear as follows: ...
1 vote
1 answer
59 views

Dealing with high frequency tokens during masked Language modelling?

Suppose I am working with a Masked Language Model to pre-train on a specific dataset. In that dataset, most sequences have a particular token of a high frequency ...
  • 167
0 votes
1 answer
50 views

There could be a problem with the linear layer after the attention inside a transformer?

My question regards this image: It seems that after the multi head attention there is a linear layer as they mention also from here: the linearity is given by the weights W^{o}. my quesion is: for ...
  • 85
1 vote
1 answer
205 views

Decoder Transformer feedforward

I have a question about the decoder transformer feed forward during training. Let's pick an example: input data "i love the sun" traduction i want to ...
  • 85
1 vote
1 answer
51 views

Anonymize continuous variable for masking purposes

I am about to kick off a large hackathon event. We have a dataset that is comprised of one continuous variable with high precision, and a number of categorical variables qualifying these data 3-levels ...
  • 901