I was trying to understand how tokens work and all I understood is that tokens are the representation of the input in a more meaningful way (data preparation for the "encoder of transformer" or "BERT").

But when i see use of special tokens like this: https://arxiv.org/pdf/2005.01107v1.pdf,

i realized that you can actually "specify" your purpose while training your data. For example, in an answer in StackOverflow, it says :

"Just an example, in extractive conversational question-answering it is not unusual to add the question and answer of the previous dialog-turn to your input to provide some context for your model. Those previous dialog turns are separated with special tokens from the current question. Sometimes people use the separator token of the model or introduce new special tokens. The following is an example with a new special token [Q]"

[CLS] previous question [Q] current question [SEP] text [EOS]

But it doesnt explain how any NLP model can use and can be trained in these tokens. How is it being training such a way that it understands that " i should be aware of previous question to answer current question" ?


1 Answer 1


The original code from the Bert model doesn't have any mention of special tokens [CLS], [SEP], or [EOS].

So it seems that the data in input is already organized to fit Bert's input format.

There is a Bert convention about those special tokens:

# The convention in BERT is:
# (a) For sequence pairs:
#  tokens:   [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
#  type_ids: 0     0  0    0    0     0       0 0     1  1  1  1   1 1
# (b) For single sequences:
#  tokens:   [CLS] the dog is hairy . [SEP]
#  type_ids: 0     0   0   0  0     0 0
# Where "type_ids" are used to indicate whether this is the first
# sequence or the second sequence. The embedding vectors for `type=0` and
# `type=1` were learned during pre-training and are added to the wordpiece
# embedding vector (and position vector). This is not *strictly* necessary
# since the [SEP] token unambiguously separates the sequences, but it makes
# it is easier for the model to learn the concept of sequences.
# For classification tasks, the first vector (corresponding to [CLS]) is
# used as the "sentence vector". Note that this only makes sense because
# the entire model is fine-tuned.

This is from the feature extraction process.

Consequently, [CLS] tokens are useful to get a row of position tokens, and [SEP] tokens are useful to differentiate the questions from answers through type_ids.

The position and type tokens are converted to tensors. However, the [CLS] and [SEP] tokens are not converted to tensors because their function is just to delimit the input data.

In the Bert model, the token_type_ids are used in the post-processor embedding, together with the position_ids to an output.

  if use_token_type:
    if token_type_ids is None:
      raise ValueError("`token_type_ids` must be specified if"
                       "`use_token_type` is True.")
    token_type_table = tf.get_variable(
        shape=[token_type_vocab_size, width],
    # This vocab will be small so we always do one-hot here, since it is always
    # faster for a small vocabulary.
    flat_token_type_ids = tf.reshape(token_type_ids, [-1])
    one_hot_ids = tf.one_hot(flat_token_type_ids, depth=token_type_vocab_size)
    token_type_embeddings = tf.matmul(one_hot_ids, token_type_table)
    token_type_embeddings = tf.reshape(token_type_embeddings,
                                       [batch_size, seq_length, width])
    output += token_type_embeddings

Then, this output is normalized and a dropout is applied.

In conclusion, special tokens are defined by a convention, and the 2 main ones are [CLS] and [SEP] which delimit the 2 main types of vectors necessary for the Bert model for the question/answer process.

Note: You can define [CLS] or [SEP] with other names in the Pretrained tokenizer from HuggingFace with the sep_token and the cls_token attributes.

  • $\begingroup$ Thanks for the answer, you said "SEP] tokens are useful to differentiate the questions from answers through type_ids" Yes, but how is this helping model to understand that "i should look paragraph and generate answers from here" ? We dont have if-else inside the model that will say: "if type_id==1 , generate questions from here" $\endgroup$
    – canP
    Commented Jul 14, 2022 at 15:36
  • 1
    $\begingroup$ There is an additional segment embedding thanks to [SEP]: Bert connects questions to answers, in addition to words or subwords. See also: towardsdatascience.com/… $\endgroup$ Commented Jul 14, 2022 at 16:46
  • $\begingroup$ Exactly, the thing that I don't know is how those segment embeddings help the model to train in such a way that it understands that "segment embedding No:1 is paragraph" ? etc. please check this question too, maybe in this link i clarified my answer better: ai.stackexchange.com/questions/36342/… $\endgroup$
    – canP
    Commented Jul 14, 2022 at 16:49
  • $\begingroup$ Sorry, I can't answer in a comment to a question asked elsewhere. $\endgroup$ Commented Jul 15, 2022 at 8:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.