Difference between using BERT as a 'feature extractor' and fine tuning BERT with its layers fixed

Question

I understand that there are two ways of leveraging BERT for some NLP classification task:

1. BERT might perform ‘feature extraction’ and its output is input further to another (classification) model
2. The other way is fine-tuning BERT on some text classification task by adding an output layer or layers to pretrained BERT and retraining the whole (with varying number of BERT layers fixed

However, if in the second case, we fix all the layers and add ALL the layers from the classification model will be added, 1st and 2nd approaches are effectively the same, am I right?

Answer 1

No, approaches 1 and 2 are not the same:

• In approach 1 (feature extraction), you not only take BERT's output, but normally take the internal representation of all or some of BERT's layers.

• In approach 2, you train not only the classification layers but all BERT's layers also. Normally, you choose a very low learning rate and a triangular learning rate schedule to avoid catastrophic forgetting.

There are many scientific articles studying how to best use BERT in transfer learning scenarios. This one may be a good starting point: https://www.aclweb.org/anthology/W19-4302/

Answer 2

In approach 2, after you have fixed all the Bert layers and trained the last classification model, you have the option to un-fix the Bert layers and further train your model, and get (hopefully ) more optimized parameters for your current task.

In Pytorch this can be done by:

first fix layers

for param in model.parameters():
    param.requires_grad = False

then unfix layers

for param in model.parameters():
    param.requires_grad = True