What do special tokens used for in Roberta?

Question

When I use this code:

from transformers import RobertaTokenizer

tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
captions = "This is an example caption"

output = tokenizer(captions, padding='longest')
input_ids = output['input_ids']
print(input_ids)

tokens = tokenizer.convert_ids_to_tokens(input_ids)
print(tokens)

The output is:

[0, 713, 16, 41, 1246, 3747, 2]  
['<s>', 'This', 'Ġis', 'Ġan', 'Ġexample', 'Ġcaption', '</s>']

What is the purpose of the special tokens <s> and </s>, are they really necessary, does the performance decrease if I remove them using add_special_tokens = False ?

Thanks

Answer 1

The special tokens <s> and </s> are indeed utilized for specific NLP tasks such as question answering, sequence classification, and language modeling. However, in your case, where the primary objective is to extract contextual embeddings from the hidden states of the RoBERTa model, the necessity of these special tokens may be influenced by the specific use case and the downstream tasks for which the embeddings will be employed.

When extracting contextual embeddings from the hidden states, the special tokens may not directly impact the process of obtaining the embeddings themselves. The hidden states contain the contextual information for each token in the input sequence, and the presence or absence of special tokens may not significantly affect the extraction of these embeddings.

It's important to note that the special tokens play a role in the overall architecture and functioning of the model, and their inclusion is part of the pre-training and fine-tuning processes. However, for the specific task of extracting contextual embeddings from the hidden states, the impact of the special tokens may be less pronounced compared to tasks directly reliant on sequence boundaries and structure, such as sequence classification or question answering.