2
$\begingroup$

I'm exploring the training efficiency of transformer models against the backdrop of data quality sequencing. Specifically, I ponder whether arranging unlabeled data by presumed quality affects training outcomes, akin to the structured progression observed in human learning paradigms. This inquiry stems from the hypothesis that a gradual escalation in data complexity or a focused emphasis on high-quality data during training (e.g., more epochs for data subsets) might optimize model performance.

I seek insights or empirical evidence from this community’s experiences or relevant literature that might illuminate the impact of data sequencing on transformer training. Any shared knowledge or references to studies exploring this facet would be invaluable.

$\endgroup$

1 Answer 1

3
$\begingroup$

From your question, it is not clear to me if you are referring to training with increasing data complexity or increasing data quality (for some definition of "quality").

If you are referring to training with increasing complexity, it has been studied and the relevant term to refer to it is curriculum learning.

In traditional training, all training samples are usually presented to the model randomly. However, in curriculum learning, the model starts with simpler or easier examples and progressively moves to more complex ones. This mimics the way humans often learn, starting with basic concepts before moving on to more advanced topics.

This has been studied for different tasks in the past (e.g. translation, NLU). In the context of LLMs, you can find results in the Orca paper and more recently in Instruction Tuning with Human Curriculum.

In general, research seems to find curriculum learning beneficial.

$\endgroup$
3
  • $\begingroup$ Thanks for pointing it out. My initial query leaned towards complexity. I appreciate the references to curriculum learning. Very helpful. Regarding data quality, this aspect also intrigues me. If there are insights or studies on how varying data quality, not just complexity, influences transformer model training, I'd be keen to know more. $\endgroup$ Mar 3 at 23:16
  • $\begingroup$ I am not aware of studies that research the effect of different degrees of data quality. Data quality is usually not defined in "gradual" terms but in binary terms: garbage vs not garbage, and garbage data is directly removed. $\endgroup$
    – noe
    Mar 4 at 6:51
  • 1
    $\begingroup$ An intermediate level between garbage data and not garbage data may be machine-translated data (aka "translationese"). A non-negligible portion of the internet may be this kind of text. $\endgroup$
    – noe
    Mar 4 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.