What does Embeddings Array Represent in BERT's Feature Extraction?

Question

I am new to academic NLP, and I had been tasked with to use BERT to extract features of a sentence.

text_input = [
  "Hello I'm a single sentence",
  "And another sentence",
  "And the very very last one",
  "My name is Aun"
  ]

I got embeddings using pipeline from huggingface:

from transformers import pipeline
feature_extraction = pipeline('feature-extraction', model="distilroberta-base", tokenizer="distilroberta-base")
features = feature_extraction(text_input)

Embeddings were multi-dimension, which I flattened and then padded to match the array with highest size. Here text_df.head():

    text_input                  text_embeddings                                 text_em_flat                        text_em_flat_pad
0   Hello I'm a single sentence [[[-0.010155443102121353, 0.07965511828660965,...   [-0.010155443102121353, 0.07965511828660965, 0...   [-0.010155443102121353, 0.07965511828660965, 0...
1   And another sentence        [[[-0.010256338864564896, 0.0948348417878151, ...   [-0.010256338864564896, 0.0948348417878151, -0...   [-0.010256338864564896, 0.0948348417878151, -0...
2   And the very very last one  [[[-0.001137858722358942, 0.09048153460025787,...   [-0.001137858722358942, 0.09048153460025787, -...   [-0.001137858722358942, 0.09048153460025787, -...
3   My name is Aun              [[[-0.0018534815171733499, 0.08652304857969284...   [-0.0018534815171733499, 0.08652304857969284, ...   [-0.0018534815171733499, 0.08652304857969284, ...

But I don't understand what each value represents in the text_embeddings. I have gone through some explanation, but don't understand if they are token level or segment level or position level embeddings for a stack of all three. Please explain. Following are the shapes for few instances:

arr_dimen(text_df['text_embeddings'][0]):    [1, 8, 768]
arr_dimen(text_df['text_embeddings'][1]):    [1, 5, 768]
arr_dimen(text_df['text_embeddings'][2]):    [1, 8, 768]
arr_dimen(text_df['text_embeddings'][3]):    [1, 7, 768]
arlen(text_df['text_em_flat'][0]):   6144
arlen(text_df['text_em_flat'][1]):   3840
arlen(text_df['text_em_flat'][2]):   6144
arlen(text_df['text_em_flat'][3]):   5376

From original paper I understand that BERT divides input in three-layers and then uses them like shown in the figure from original paper.

But I want to understand how BERT encodes My name is Aun to an array with shape [1, 7, 768].

Answer 1

BERT computes a vector per input token. When the text is processed, it gets tokenized first. Your sentence gets tokenized like this:

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
tokenzier.tokenize("My name is Aun")

which gives 5 tokens: ['My', 'name', 'is', 'Au', '##n'] BERT uses 30k WordPiece vocabulary. The most frequent words remain intact, and less frequent words (such as Aun) get split into smaller pieces, eventually down to characters. This way, it ensures it can represent every word.

As you can see in the Figure that you copied from the BERT paper, there are two technical tokens added: [CLS] and [SEP]; together with the 5 input tokens, it gets 7 output states.

You can use batches of more sentences. Batch size is the first number of the output tensor dimension: you input just one sentence, therefore 1. (Note that with multiple sentences, the tensor will be padded to the maximum length.) The last number, 768, is the dimension of BERT's hidden states. This was an arbitrary design decision of the model authors.

The individual dimensions have no straightforward interpretation. For more about BERT's interpretability, I can refer you to a paper called A Primer in BERTology.