I am new to DataScience and trying to implement BERT embedding for one of my problems. But I am having one doubt here. I am trying to embed the following sentence with BERT - "Twinkle twinkle little star". BERT tokenizer generates the following tokens - ['twin', '##kle', 'twin', '##kle', 'little', 'star']

But the final embedded tensor is having a dimension of [1,8,1024]

Why is the number of tokens 8 instead of 6? For any text, I am observing that number of tokens in the final embedding is getting increased by 2. Can anyone please help me to understand this?

I am giving the code snippet here -

from transformers import BertTokenizer, BertForSequenceClassification, BertModel

PRE_TRAINED_MODEL_PATH = 'BERT\wwm_cased_L-24_H-1024_A-16'
tokenizer = BertTokenizer.from_pretrained(PRE_TRAINED_MODEL_PATH)
model = BertModel.from_pretrained(PRE_TRAINED_MODEL_PATH)

encoded_input = tokenizer(texts, return_tensors='pt', padding=True)
emb = model(**encoded_input)

A [CLS] token is added to the beginning of the sentence, and a [SEP] token is added to the end.

These two special tokens have specific functions:

  • [CLS] is needed because it was used in the training loss of BERT to keep the first output position for a different purpose than the rest: it was used for the "next sentence prediction" loss, which trained the model to tell if the 2 text segments passed as input were consecutive in the original text or not. When finetuning BERT, the output of this position is normally used for sentence classification tasks. When BERT is used for feature extraction, the output vector at that position is used as sentence embedding.

  • [SEP] is used to mark the ending of the sentence.

  • $\begingroup$ Thanks for the answer. So, the cases where I am using BERT for NER - can we remove [CLS] and [SEP] from the embedding - as there will be no corresponding NER tags in our dataset. $\endgroup$ – Saikat Bhattacharya Jan 8 at 8:23
  • $\begingroup$ This completely depends on how you are using those embedded vectors, e.g. does your network need a marker for the end of the sentence? Probably yes, so maybe it's safer for you to keep the [SEP] token. For the [CLS] token, I would say it's safer to drop it. Anyway, I suggest you evaluate the effect of having or not having them in your final performance. $\endgroup$ – noe Jan 8 at 8:54
  • $\begingroup$ Thank you. I get the point. $\endgroup$ – Saikat Bhattacharya Jan 8 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.