I want to get sentence embeddings of transformer-based models (Bert, Roberta, Albert, Electra...).

I plan on doing mean pooling on the hidden states of the second last layer just as what bert-as-service did.

So my questions is that when I do mean pooling, should I include the embeddings related to [PAD] tokens or [CLS] token or [SEP] token?

For example, my sequence is 300 tokens, and are padded into 512 tokens.

The output size is 512 (tokens) * 768 (embeddings).

So should I average the embeddings of first 300 tokens or the embeddings of whole 512 tokens?

Why the embeddings of the last 212 tokens are non-zero?


1 Answer 1


Ques 1. You decide how you want your padded pooling layer to behave.This is why pytorch's avg pool (e.g., nn.AvgPool2d) has an optional parameter count_include_pad=True: By default (True) Avg pool will first pad the input and then treat all elements the same. On the other hand, if you set count_include_pad=False the pooling layer will ignore the padded elements and the result

When you have too many padding tokens they take part in gradient calculation as part of training and weights updates they gain some weights. Obviously these weights computation are waste of time and should be avoided.

Suggestion 1: Make sure every watch has similar size sentences to avoid a lot of padding column


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.