When you run BERT, you get one vector per input token + 1 special token called
[CLS] + 1 special token called
[SEP]. Maybe more precise than calling BERT embeddings as embeddings, would be calling them hidden states of BERT. The contextual information get into the embeddings via 12 layers of self-attentive neural network.
However, the tokenization is tricky with BERT, the tokens are not words. It uses so-called WordPieces to represent the input, i.e., less frequent words are split into smaller units, so at the end, there are no OOV tokens.
With the BERT Base Cased model, the tokenization looks like this:
'I am the walrus.' →
['I', 'am', 'the', 'wa', '##l', '##rus', '.']
'What are the elements in a BERT word embedding?' →
['What', 'are', 'the', elements', 'in', 'a', 'B', '##ER', '##T', 'word', 'em', '##bed', '##ding', '?']
When BERT is trained, there are always two sentences, separated by the
[SEP] token. The embedding of the
[CLS] is used to predict if the two sentences follow each other in a coherent text. In sentence classification tasks, the embedding of the
[CLS] token is used as an input to the classifier.