Say we have a fully-trained N
-layer transformer model (encoder-only, decoder-only, or encoder-decoder), with embedding dimension D
, trained for E
epochs on S
training strings.
Then we take one of those layers, layer L
, and reset it back to random weights (all of it, the self-attention weight, the FFN weights, even the layer norm parameters). Assume weights in all other layers are frozen.
What I'm wondering is how quickly it can re-learn. Does it need to see all S x E
training examples all over again? Or will it be much quicker, perhaps just needing to see 1% or 10% of the original training data? Or maybe it will go the other way, and refuse to train because the frozen weights around it stop it finding a minima??
I'm also curious if where L
is in the stack makes much difference, and if N
and D
are factors, or the result is about the same however big the transformer is.
(By "re-learn" I mean get back to approximately the same loss/perplexity on validation data that the model had before; I do not mean recreate the exact same weights.)
I'm looking for answers that point to existing papers that have done these experiments, or equally your personal experiments. Or even a "no-one has ever tried this, we don't know", if you think you can say that with confidence.
Aside: I did a test last year of swapping in a different encoder (of different dimension), in an encoder-decoder model, and wiping out the cross-attention weights, but otherwise not touching the rest of the decoder. It acted as if the whole model had been reset to random weights, and seemed to need retraining from scratch, even though 90+% of it was still "trained". But that was only a quick test. Before trying again I'd like to have a better idea of how much retraining time might be needed. (And it struck me that wiping out a layer rather than all cross-attention weights might be a cleaner experiment to learn from.)