Pretrained transformers (GPT2, Bert, XLNET) are popular and useful because of their transfer learning capabilities.
Just as a reminder: The goal of Transfer learning is is to transfer knowledge gained from one domain/task and use that transfer/use that knowledge to solve some related tasks. This is done by training a model on a huge amount of labelled data (which we already have and is probably easy to get), then remove the last few layers and fine-tune the model for the new related task with task-related dataset.
I took a recent pretrained transformer published by Google, called XLNET, and just added the classification layer from the top of it and fine-tuned the whole network. (Which is the main intention of this kind of model, correct me if I am wrong).
The problem is that the model is largely overfitting. I have 1200 examples to train and each has 350 words on average.
To overcome the overfitting, I set the dropout of each layer of the transformer from 0.1 to 0.5. This did not work. So I decreased the number of trainable parameters (since the transformer has a huge number of parameters), by freezing first 10 layers (11 layers + 1 classification layer in total). Even that does not work. So I counted the number of trainable parameters in the last layer. There are 7680000 parameters which are very high compared to my dataset (around 1200*350= 420000 words). So, this high number of tunable parameters is the most possible reason for overfitting.
Here is the loss graph:
My questions are: Do you see any flaw in my analysis? Is there anything I can do to decrease overfitting? (tried with low learning rate and large batch size) If my analysis is correct, then the claim that "fine-tune pre-trained transformers with small dataset" is bit misleading and datasets should not be so small. Am I correct?