Can a Transformers be used for a classification problem?

Question

I want to perform binary classification on a sequential data. I want to leverage the PyTorch nn.TransformerEncoder module, is there a way I can do this? If not, are Transformers only used NLP? If yes, how can I format the inputs and what tutorial can help me do this as quick as possible

Answer 1

Transformers can be used for classification tasks. I found a good tutorial where they used a BERT Transformer for the encoding and a Convolutional Neural Network for a sentiment analysis.

You can also fine-tune a whole Transformer for the classification but this is usually pretty intense when it comes to the training and you definitely need a GPU.