I am trying to fine-tune a Bert model for sentiment analysis. Instead of one sentence, my inputs are documents (including several sentences) and I am not removing dots. I was wondering if is it okay to use just the embedding of the first token in such cases. If not, what should I do?
2 Answers
$\begingroup$
$\endgroup$
Yes, it's perfectly fine to fine-tune BERT on sequences comprised of more than one sentence, and the standard way of using BERT for text classification is with the ouput vector at the first position.
However, take into account that the maximum length of BERT's input sequences is 512 tokens, so your documents should be short enough to fit in that.
$\begingroup$
$\endgroup$
Consider using other models that are Bert-based, those models often are more developed in terms of the final applications. For example, this is the open-source Bert-based encoder classification model with a 100k token capacity, so it might be a great case for extensive document processing.
