Smart Selection of Training Data for Fine-tuning Language Models in Small Domains

Question

Background

I am working to make language models (for example, Stanford's Alpaca model) perform well on a new small domain through fine-tuning on domain-specific dataset $D$.

If the size of $D$ is $N$, I would like to find a subset $n \ll N$ to fine-tune the model due to my limited computation resource: I could only afford to fine-tune the model for 24 GPU hours but fine-tuning on the entire $D$ will take 2400 GPU hours.

Question

Is there any strategy I could select $n$ smartly so that the model I fine-tuned is likely to perform better than if I choose $n$ in alternative ways? Here are two options I could think of:

• Randomly select $n$ from $N$.
• Measure the quality of each of $N$ in some way and fine-tune the data by selecting those of higher quality ones. I got this idea from curriculum learning (a survey paper in 2020).

Note

This question has been cross-posted from CrossValidated.

Answer 1

You can use Data Selection with Importance Resampling (DSIR), which applies importance resampling with bag-of-words ngrams estimators. This method is specifically meant to select good data for LLM training.

The authors of the method released their source code, so you can just use it.

This Twitter thread is a good summary.