What is the effect of the tokens that the model has if model A has 1B tokens and the other model has 12B tokens? Will that have an effect on the performance?
1 Answer
$\begingroup$
$\endgroup$
3
The question is not precise enough, it depends on other factors: in general, a larger training set tends to lead to a better model. However it depends if the training set is really relevant and useful for the task. For example:
- adding the larger dataset contains data from a different domain than the target task, the additional data might be useless
- if the data contains a lot of errors or noise, it might cause the model to perform worse
- if the larger data contains mostly duplicates, it's likely not to perform better.
So larger data is good for performance only if the additional data is actually of good quality.
-
$\begingroup$ thanks for the answer but I mean if I have three different models and I need to use them for a task and each one has a different number of tokens like in the question .. so is it a rule that the model which has large tokens means more efficient than the others .. that was my question $\endgroup$– LeiCommented Jul 21, 2022 at 8:00
-
$\begingroup$ @Lei well as I said it's not so simple, there's definitely no such rule. It's possible that the larger model would perform better, but it's not sure. $\endgroup$– ErwanCommented Jul 21, 2022 at 10:29
-
$\begingroup$ Appreciate your time and the answer .. Thanks a lot $\endgroup$– LeiCommented Jul 21, 2022 at 11:13