I am new to DataScience and trying to implement BERT embedding for one of my problems. But I am having one doubt here. I am trying to embed the following sentence with BERT - "Twinkle twinkle little star". BERT tokenizer generates the following tokens - ['twin', '##kle', 'twin', '##kle', 'little', 'star']
But the final embedded tensor is having a dimension of [1,8,1024]
Why is the number of tokens 8 instead of 6? For any text, I am observing that number of tokens in the final embedding is getting increased by 2. Can anyone please help me to understand this?
I am giving the code snippet here -
from transformers import BertTokenizer, BertForSequenceClassification, BertModel PRE_TRAINED_MODEL_PATH = 'BERT\wwm_cased_L-24_H-1024_A-16' tokenizer = BertTokenizer.from_pretrained(PRE_TRAINED_MODEL_PATH) model = BertModel.from_pretrained(PRE_TRAINED_MODEL_PATH) encoded_input = tokenizer(texts, return_tensors='pt', padding=True) emb = model(**encoded_input)