Open-Source Large Language Models (LLM): Your experience and recommendation

Question

I’m looking for an open-source LLM for a new project. I want to use it for instructions and to fine-tune the model to a specific domain like legal and rights. Some LLMs are open-source, but they didn’t document, on which training data they trained their model. This makes it a bit complicated in my case.

I’m looking for models, that are open-source and the community knows on which datasets the model was trained.

Do you know open-source LLMs like that and do you have experience with them?

Thank you in advance.

Answer 1

You should check the Open LLM Leaderboard from Huggingface.

They maintain a ranking of open LLMs in different tasks. The ones finetuned to follow instructions are marked. When clicking on each model, you go to the model card, which contains details about the model and the data it was trained with.

However, I think your requirement of "know[ing] on which datasets the model was trained" is difficult to meet. These models are trained on vasts amounts of data, so normally the training data is "everything the authors could get their hands on". Because of this, in many cases there is no know training dataset for LLMs.

An exception I know of is BLOOMZ, which finetuned BLOOM (train on the [ROOTS dataset])(https://arxiv.org/abs/2303.03915) on the xP3 instruction-following dataset.