LMM Fine Tuning - Supervised Fine Tuning Trainer (SFTTrainer) vs transformers Trainer

Question

When should one opt for the Supervised Fine Tuning Trainer (SFTTrainer) instead of the regular Transformers Trainer when it comes to instruction fine-tuning for Language Models (LLMs)? From what I gather, the regular Transformers Trainer typically refers to unsupervised fine-tuning, often utilized for tasks such as Input-Output schema formatting after conducting supervised fine-tuning. There seem to be various examples of fine-tuning tasks with similar characteristics, but with some employing the SFTTrainer and others using the regular Trainer. Which factors should be considered in choosing between the two approaches?

I looking for Fine Tuning a LLM for generating json to json transformation (matching texts in json) using huggingface and trl libraries.

Answer 1

The short answer is that a Supervised Fine Tuning Trainer (SFTTrainer) is used for Instruct Fine Tuning. The HuggingFace library SFTTrainer has also support for training with QLoRA (4-bit Quantised model forward pass and LoRA adapters), and also saving the model with that. From the source code the actual work is done by the Trainer baseclass. Note that both the classes are Supervised in that they use the next label (Causal LM ) as the target.

Instruction Tuning concept is a higher-level training concept introduced by this paper FineTuned Language Models Are Zero shot Learners (FLAN).

Example dataset for the QloRa paper implementation - https://huggingface.co/datasets/mlabonne/guanaco-llama2/viewer/default/train?row=0