I want to get sentence embeddings of transformer-based models (Bert, Roberta, Albert, Electra...).
I plan on doing mean pooling on the hidden states of the second last layer just as what bert-as-service did.
So my questions is that when I do mean pooling, should I include the embeddings related to [PAD] tokens or [CLS] token or [SEP] token?
For example, my sequence is 300 tokens, and are padded into 512 tokens.
The output size is 512 (tokens) * 768 (embeddings).
So should I average the embeddings of first 300 tokens or the embeddings of whole 512 tokens?
Why the embeddings of the last 212 tokens are non-zero?