I started to work with LLMs lately and want to know how people choose their pre-trained models in their fine-tuning tasks? What is the criteria to choose the base model and which factors affect?
2 Answers
$\begingroup$
$\endgroup$
There are too many! Some examples:
Intended use of the model regarding its compatibility with the licenses of available model (p.ej. is commercial use allowed?).
intended use of the model to know if the model should be instruction-tuned or not.
memory and compute restrictions (e.g. deploying on a mobile device vs. deploying on a huge cloud machine with dozens of GPUs).
languages to support.
potential size of the context window.
Domain-specific issues (p.ej. better to use a model trained on legal texts if you are going to use it for the legal domain).
$\begingroup$
$\endgroup$
Things to consider for choosing a base model:
The size of the model is a crucial starting point - while larger models generally perform better, you need to balance this with your computational resources and deployment constraints. Consider whether you can effectively train and run the model within your infrastructure.
Then there's domain relevance - the pre-trained model's training data should align with your target domain. For instance, if you're working on biomedical applications, models pre-trained on scientific literature like PubMedBERT might be more suitable than general-purpose models.
Cost is another practical factor - both in terms of computational resources for fine-tuning and inference costs in production. Smaller models might be more cost-effective for your specific use case.
The licensing terms of the model are crucial too - make sure the model's license allows for your intended use case, whether commercial or research. Finally, consider the model's documented performance on tasks similar to yours, along with its stability and community support. A well-documented model with active community support can save you significant debugging time.
