Further Training a pre-trained LLM

Question

My goal is to use the general knowledge and language understanding of a pre-trained LLM and to continue training on a smaller domain specific corpus to improve the model's knowledge on the domain. What is the best practice approach here without running into issues (e.g. catastrophic forgetting)? Here are some points I consider, but not completely sure about them:

• use last checkpoint of pre-trained LLM and continue training on custom corpus
• training policy and procedure is the same as used for pre-training (MLM etc.)
• use a very small learning rate
• is it possible to load the model in int8 (bitsandbytes) and continue training without breaking it?

Does this approach make sense? Has anyone done this before and has some insights?

Any hints are highly appreciated!

Answer 1

Yes you are on the right track. What you are mentioning is called fine tuning the model. I personally have done this and used the same approach. The LLM I used was GPT-J 6B to generate MCQ's. Some tips when fine tuning large LLM's are:

1. Do not feed all the data to the model. First create a small dataset and fine tune the model in order to check whether the fine tuning is working properly for a couple epochs. This may save you some hours down the road.
2. Make sure you understand all the Hyperparameters and their effects first before fine tuning. Since the model is large, fine tuning might take a long time and resources and you do not want to fine tune multiple times with different parameters. So save yourselves some time and money by understanding the effects of the parameters first.
3. Yes it is possible to load the model into lower bits (also known as quantisation) in order to reduce the model size and in turn reduce resource utilisation. Make sure to quantise before training and then before inferencing, make sure to de-quantise the model to original bits. This is important as I was getting nonsense results just because I forgot to de-quantise the model after training and before inferencing.
4. Your first point of only using the last layer. I am not sure about this as I did not try this method.
5. Regarding the 3rd point of using a very small learning rate, it is a hyperparameter. So you might want to tune it. But usually if the model is huge (as in my case 6 billion parameters, 100 GB) even if there is some leeway in tuning the parameters, it won't affect the results much as the model is robust enough to counter it. But again it depends on your model size!

Cheers!

Answer 2

I find this link very relevant. Basically, you need to find the right AutoModel type for your model-specific word prediction task.

Answer 3

I know this question was asked months ago, but just wanted to chime in in case this link is useful: https://huggingface.co/learn/nlp-course/chapter7/3?fw=pt

Shows how to fine-tune a masked language model. Not sure what your goal is for a downstream task, but I ended up using a base BERT model, then "fine-tuned" it on new domain specific data with a masked language approach. Essentially, I considered this continued training since I had no intention of using the model for masked language tasks. Once you've continued training, you have a better base model you can fine-tune for the task you are actually interested in. I used this new BERT model for sentiment and sentence similarity after fine-tuning for those tasks...

Oh, just notice that you are interested in using the model for named entry recognition. BERT models are suited for that task. Another advantage of a base BERT model is that if you want to feed it heaps of data it's pretty easy to do since all you are doing in the continued training is masking x percent of the words in the text.