Why Transformer applies Dropout after Positional Encoding?

Question

Why Transformers applies Dropout after Positional Encoding?

Attention Is All You Need

Not sure what is the benefit of removing 10% of tokens in a sequence by default. Read Why use dropout in positional encoding layer but not clear.

Also does the PyTorch implementation of PositionalEncoding with Dropout may drop [CLS] or [SEP]?

Pytorch class PositionalEncoding(nn.Module):

class PositionalEncoding(nn.Module):

    def __init__(self, d_model: int, dropout: float = 0.1, max_len: int = 5000):
        super().__init__()
        self.dropout = nn.Dropout(p=dropout)

        position = torch.arange(max_len).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2) * (-math.log(10000.0) / d_model))
        pe = torch.zeros(max_len, 1, d_model)
        pe[:, 0, 0::2] = torch.sin(position * div_term)
        pe[:, 0, 1::2] = torch.cos(position * div_term)
        self.register_buffer('pe', pe)

    def forward(self, x: Tensor) -> Tensor:
        """
        Arguments:
            x: Tensor, shape ``[seq_len, batch_size, embedding_dim]``
        """
        x = x + self.pe[:x.size(0)]
        return self.dropout(x)         # <---- May drop [CLS] or [SEP]?

Answer 1

Normal dropout does not remove whole tokens, but individual values within the vectors. Therefore, dropout does not remove 10% of the tokens in a sequence, but 10% of the values.

There is a different type of dropout called "token dropout" that works at token level and therefore works as you described. Such type of dropout, however, is not the one described in the paragraph you highlighted.

Update: Given that a user showed doubts about my answer, here is a piece of code to show the effects of nn.Dropout after an embedding layer. Feel free to run it in Google colab to check its effect by yourselves:

import torch
import torch.nn as nn

# Set seed for reproducibility
torch.manual_seed(42)

# Define parameters
vocab_size = 10
embedding_dim = 5
dropout_prob = 0.5  # Dropout probability

# Create embedding layer
embedding = nn.Embedding(vocab_size, embedding_dim)

# Create dropout layer
dropout = nn.Dropout(p=dropout_prob)

# Input tensor
input_tensor = torch.LongTensor([[1, 2, 3]])

# Apply embedding layer
embedded_tensor = embedding(input_tensor)

# Apply dropout layer
dropout_tensor = dropout(embedded_tensor)

# Print original and dropout tensor
print("Original Tensor:")
print(embedded_tensor)
print("\nTensor after Dropout:")
print(dropout_tensor)

Here is the output (take into account that, apart from zeroing some positions, the outputs are scaled by a factor of $\frac{1}{1-p}$):

Original Tensor:
tensor([[[-1.2345, -0.0431, -1.6047, -0.7521,  1.6487],
         [-0.3925, -1.4036, -0.7279, -0.5594, -0.7688],
         [ 0.7624,  1.6423, -0.1596, -0.4974,  0.4396]]],
       grad_fn=<EmbeddingBackward0>)

Tensor after Dropout:
tensor([[[-0.0000, -0.0861, -0.0000, -0.0000,  0.0000],
         [-0.7850, -0.0000, -0.0000, -0.0000, -0.0000],
         [ 0.0000,  3.2846, -0.3192, -0.9948,  0.0000]]],
       grad_fn=<MulBackward0>)