How can I use contextual embeddings with BERT for sentiment analysis/classification

Question

I have a BERT model which I want to use for sentiment analysis/classification. E.g. I have some tweets that need to get a POSITIVE,NEGATIVE or NEUTRAL label. I can't understand how contextual embeddings would help in a better model, practically.

I process the tweets and sentences to make them ready to be fed into the tokenizer. After I get every embedding as well as its mask, and feed it into a BERT model. From that BERT model I get some hidden states in return. As I understand it, now I have to also use a linear layer to take that 768 output from BERT and output a possibility for the 3 labels.

How can contextual embeddings help me here? I get that we can use a combination of those hidden states/layers that we get for every sentence by the BERT model, and that helps us create better embeddings, which technically mean better models. But, after I follow some approach, e.g. summing the last four hidden states, or taking a mean of every token to create a token for each word, how do I proceed now? Do I need another model to take that embedding and output the labels that way (e.g. a linear layer but after the contextual embeddings are created)? Am I thinking of this the right way? Any input would be appreciated.

Answer 1

For this kind of setup, you should use the output at the first position and train a linear classifier over your 3 labels.

BERT was trained with inputs that were prepended a special token [CLS], and the output at that position (i.e. the first position) was used for a classification task. It is understood that, at that position, BERT outputs a representation for the whole input sentence. Therefore, this representation (i.e. the vector with dimensionality 768 outputted by the last layer in the first position) is what you should use as input for your sentiment classifier.

You should train a new model that takes the 768-dimensionality vector as input and generates the label. For this, a linear model with a categorical cross-entropy loss would be the standard.