# Why does BERT embedding increase the number of tokens?

I am new to DataScience and trying to implement BERT embedding for one of my problems. But I am having one doubt here. I am trying to embed the following sentence with BERT - "Twinkle twinkle little star". BERT tokenizer generates the following tokens - ['twin', '##kle', 'twin', '##kle', 'little', 'star']

But the final embedded tensor is having a dimension of [1,8,1024]

Why is the number of tokens 8 instead of 6? For any text, I am observing that number of tokens in the final embedding is getting increased by 2. Can anyone please help me to understand this?

I am giving the code snippet here -

from transformers import BertTokenizer, BertForSequenceClassification, BertModel

PRE_TRAINED_MODEL_PATH = 'BERT\wwm_cased_L-24_H-1024_A-16'
tokenizer = BertTokenizer.from_pretrained(PRE_TRAINED_MODEL_PATH)
model = BertModel.from_pretrained(PRE_TRAINED_MODEL_PATH)

emb = model(**encoded_input)


A [CLS] token is added to the beginning of the sentence, and a [SEP] token is added to the end.
• [CLS] is needed because it was used in the training loss of BERT to keep the first output position for a different purpose than the rest: it was used for the "next sentence prediction" loss, which trained the model to tell if the 2 text segments passed as input were consecutive in the original text or not. When finetuning BERT, the output of this position is normally used for sentence classification tasks. When BERT is used for feature extraction, the output vector at that position is used as sentence embedding.
• [SEP] is used to mark the ending of the sentence.
• This completely depends on how you are using those embedded vectors, e.g. does your network need a marker for the end of the sentence? Probably yes, so maybe it's safer for you to keep the [SEP] token. For the [CLS] token, I would say it's safer to drop it. Anyway, I suggest you evaluate the effect of having or not having them in your final performance. – noe Jan 8 at 8:54