I have a BERT model which I want to use for sentiment analysis/classification. E.g. I have some tweets that need to get a POSITIVE,NEGATIVE or NEUTRAL label. I can't understand how contextual embeddings would help in a better model, practically.

I process the tweets and sentences to make them ready to be fed into the tokenizer. After I get every embedding as well as its mask, and feed it into a BERT model. From that BERT model I get some hidden states in return. As I understand it, now I have to also use a linear layer to take that 768 output from BERT and output a possibility for the 3 labels.

How can contextual embeddings help me here? I get that we can use a combination of those hidden states/layers that we get for every sentence by the BERT model, and that helps us create better embeddings, which technically mean better models. But, after I follow some approach, e.g. summing the last four hidden states, or taking a mean of every token to create a token for each word, how do I proceed now? Do I need another model to take that embedding and output the labels that way (e.g. a linear layer but after the contextual embeddings are created)? Am I thinking of this the right way? Any input would be appreciated.


1 Answer 1


For this kind of setup, you should use the output at the first position and train a linear classifier over your 3 labels.

BERT was trained with inputs that were prepended a special token [CLS], and the output at that position (i.e. the first position) was used for a classification task. It is understood that, at that position, BERT outputs a representation for the whole input sentence. Therefore, this representation (i.e. the vector with dimensionality 768 outputted by the last layer in the first position) is what you should use as input for your sentiment classifier.

You should train a new model that takes the 768-dimensionality vector as input and generates the label. For this, a linear model with a categorical cross-entropy loss would be the standard.

  • $\begingroup$ Thanks for input! Same think would apply If I wanted to use the contextual embeddings though, right? I mean, I would create a embedding representing a whole tweet for example by summing the tokens of the tweets (if I chose to do it that way, instead of using [CLS]) , and then feed it into a linear model again? Or am I missing something? Thanks again. $\endgroup$ Mar 4 at 10:04
  • 1
    $\begingroup$ Yes, you could also do it by averaging the contextual embeddings of all tokens and using the result as input to your classifier. $\endgroup$
    – noe
    Mar 4 at 10:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.