Why Transformers applies Dropout after Positional Encoding?

Attention Is All You Need enter image description here

Not sure what is the benefit of removing 10% of tokens in a sequence by default. Read Why use dropout in positional encoding layer but not clear.

enter image description here

Also does the PyTorch implementation of PositionalEncoding with Dropout may drop [CLS] or [SEP]?

Pytorch class PositionalEncoding(nn.Module):

class PositionalEncoding(nn.Module):

    def __init__(self, d_model: int, dropout: float = 0.1, max_len: int = 5000):
        self.dropout = nn.Dropout(p=dropout)

        position = torch.arange(max_len).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
        pe = torch.zeros(max_len, 1, d_model)
        pe[:, 0, 0::2] = torch.sin(position * div_term)
        pe[:, 0, 1::2] = torch.cos(position * div_term)
        self.register_buffer('pe', pe)

    def forward(self, x: Tensor) -> Tensor:
            x: Tensor, shape ``[seq_len, batch_size, embedding_dim]``
        x = x + self.pe[:x.size(0)]
        return self.dropout(x)         # <---- May drop [CLS] or [SEP]?

1 Answer 1


Normal dropout does not remove whole tokens, but individual values within the vectors. Therefore, dropout does not remove 10% of the tokens in a sequence, but 10% of the values.

There is a different type of dropout called "token dropout" that works at token level and therefore works as you described. Such type of dropout, however, is not the one described in the paragraph you highlighted.

Update: Given that a user showed doubts about my answer, here is a piece of code to show the effects of nn.Dropout after an embedding layer. Feel free to run it in Google colab to check its effect by yourselves:

import torch
import torch.nn as nn

# Set seed for reproducibility

# Define parameters
vocab_size = 10
embedding_dim = 5
dropout_prob = 0.5  # Dropout probability

# Create embedding layer
embedding = nn.Embedding(vocab_size, embedding_dim)

# Create dropout layer
dropout = nn.Dropout(p=dropout_prob)

# Input tensor
input_tensor = torch.LongTensor([[1, 2, 3]])

# Apply embedding layer
embedded_tensor = embedding(input_tensor)

# Apply dropout layer
dropout_tensor = dropout(embedded_tensor)

# Print original and dropout tensor
print("Original Tensor:")
print("\nTensor after Dropout:")

Here is the output (take into account that, apart from zeroing some positions, the outputs are scaled by a factor of $\frac{1}{1-p}$):

Original Tensor:
tensor([[[-1.2345, -0.0431, -1.6047, -0.7521,  1.6487],
         [-0.3925, -1.4036, -0.7279, -0.5594, -0.7688],
         [ 0.7624,  1.6423, -0.1596, -0.4974,  0.4396]]],

Tensor after Dropout:
tensor([[[-0.0000, -0.0861, -0.0000, -0.0000,  0.0000],
         [-0.7850, -0.0000, -0.0000, -0.0000, -0.0000],
         [ 0.0000,  3.2846, -0.3192, -0.9948,  0.0000]]],
  • $\begingroup$ Thanks for the answer. $\endgroup$
    – mon
    Commented Mar 19 at 21:01
  • $\begingroup$ I doubt that's the case. @noe can you please refer to the PyTorch implementations of the "normal dropout" and "token dropout"? $\endgroup$
    – mausamsion
    Commented Mar 20 at 6:43
  • $\begingroup$ @mausamsion AFAIK there is no standard token dropout implementation in Pytorch, one has to implement it themselves. Here is the article that proposed token dropout (see section 3.2); later, more token dropout strategies were proposed here. About the normal dropout, you can check the Pytorch docs, which specify During training, randomly zeroes some of the elements of the input tensor with probability p. $\endgroup$
    – noe
    Commented Mar 20 at 7:59
  • $\begingroup$ @mausamsion I added a small snippet to prove that the normal dropout behaves as I described in my answer. Please, let me know if your doubt has been cleared. $\endgroup$
    – noe
    Commented Mar 20 at 12:30
  • $\begingroup$ @noe thanks for the update and the code snippet. I was also trying to play with the PyTorch's dropout function. And yes, as you said it drops out some values inside the tensor. But, if I am correct, the essence of dropout (from the original paper) is to black out the whole neuron in the NN. Each neuron has a vector of values, for example if the input_seq_len is 128, there will be 128 neurons in the input layer each one with an embedding vector of the corresponding token. So, blacking out the neuron should mean zeroing the whole vector. Will follow the links you gave to read more. $\endgroup$
    – mausamsion
    Commented Mar 22 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.