1
$\begingroup$

tl;dr What are the :0 and :1 in the following huggingface processors reference usage given on their page:

tokenizer.post_processor = processors.TemplateProcessing(
    single=f"[CLS]:0 $A:0 [SEP]:0",
    pair=f"[CLS]:0 $A:0 [SEP]:0 $B:1 [SEP]:1",
    special_tokens=[("[CLS]", cls_token_id), ("[SEP]", sep_token_id)],
)

Here the cls_token_id is 2 and sep_token_id is 3

Description: Post you train a tokenizer using the huggingface library you have to then add a postprocessr for things like adding the [CLS] token at the beginning of every sentence.

The following is an example from their documentation

enter image description here

Can you help me understand what purpose are the :0 and :1 serving?

$\endgroup$

1 Answer 1

2
$\begingroup$

The answer is in your screenshot:

For each of these (special tokens and sentences), we also specify the corresponding token type ID after a colon

Therefore, the :0 marks the associated tokens to have token ID 0, while :1 marks the associated tokens to have token ID 1.

Token IDs are a generic concept to mark "something else" apart from the tokens themselves. They are usually used to mark the tokens belonging to the first and second segments when the input is comprised of both of them. In the original BERT article, these are referred to as "segment embeddings": (where Eᴀ equates to ":0" in huggingface output, and Eʙ equatest to ":1")

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.