My goal is to use the general knowledge and language understanding of a pre-trained LLM and to continue training on a smaller domain specific corpus to improve the model's knowledge on the domain. What is the best practice approach here without running into issues (e.g. catastrophic forgetting)? Here are some points I consider, but not completely sure about them:
- use last checkpoint of pre-trained LLM and continue training on custom corpus
- training policy and procedure is the same as used for pre-training (MLM etc.)
- use a very small learning rate
- is it possible to load the model in int8 (bitsandbytes) and continue training without breaking it?
Does this approach make sense? Has anyone done this before and has some insights?
Any hints are highly appreciated!