1
$\begingroup$

When I use this code:

from transformers import RobertaTokenizer

tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
captions = "This is an example caption"

output = tokenizer(captions, padding='longest')
input_ids = output['input_ids']
print(input_ids)

tokens = tokenizer.convert_ids_to_tokens(input_ids)
print(tokens)

The output is:

[0, 713, 16, 41, 1246, 3747, 2]  
['<s>', 'This', 'Ġis', 'Ġan', 'Ġexample', 'Ġcaption', '</s>']

What is the purpose of the special tokens <s> and </s>, are they really necessary, does the performance decrease if I remove them using add_special_tokens = False ?

Thanks

$\endgroup$
2
  • $\begingroup$ Those tokens are the start and end tokens, and define the start and the end of the output. If no output token is generated the model will continue to generate output. This answer on stackoverflow gives some more information on what special tokens are. $\endgroup$
    – Oxbowerce
    Feb 24 at 13:25
  • $\begingroup$ Thanks, from what I read the special tokens are used for question answer, sequence classification, etc. But I don't use these tasks, I just extract the hidden states to get the contextual embeddings ? $\endgroup$
    – user159173
    Feb 24 at 13:32

1 Answer 1

1
$\begingroup$

The special tokens <s> and </s> are indeed utilized for specific NLP tasks such as question answering, sequence classification, and language modeling. However, in your case, where the primary objective is to extract contextual embeddings from the hidden states of the RoBERTa model, the necessity of these special tokens may be influenced by the specific use case and the downstream tasks for which the embeddings will be employed.

When extracting contextual embeddings from the hidden states, the special tokens may not directly impact the process of obtaining the embeddings themselves. The hidden states contain the contextual information for each token in the input sequence, and the presence or absence of special tokens may not significantly affect the extraction of these embeddings.

It's important to note that the special tokens play a role in the overall architecture and functioning of the model, and their inclusion is part of the pre-training and fine-tuning processes. However, for the specific task of extracting contextual embeddings from the hidden states, the impact of the special tokens may be less pronounced compared to tasks directly reliant on sequence boundaries and structure, such as sequence classification or question answering.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.