Best practice for fine tuning LLM

Question

I have a dataset that I have collected for specific topic.

The dataset is in the following format:

1. Raw text (similar to shakespeare dataset) where it has no label or input, just text
2. Question and answer dataset similar to alpaca instructs

My way is to fine tune a LLM on raw text first then on Q&A dataset

Does this looks like good approach? Or can I just fine tune it on all the datasets together?

Answer 1

yes you are going in the write path - 1 step - sequential Fine Tuning 2 step - Simultaneous Fine-Tuning 3 step - Recommendation