0
$\begingroup$

What is the effect of the tokens that the model has if model A has 1B tokens and the other model has 12B tokens? Will that have an effect on the performance?

$\endgroup$

1 Answer 1

0
$\begingroup$

The question is not precise enough, it depends on other factors: in general, a larger training set tends to lead to a better model. However it depends if the training set is really relevant and useful for the task. For example:

  • adding the larger dataset contains data from a different domain than the target task, the additional data might be useless
  • if the data contains a lot of errors or noise, it might cause the model to perform worse
  • if the larger data contains mostly duplicates, it's likely not to perform better.

So larger data is good for performance only if the additional data is actually of good quality.

$\endgroup$
3
  • $\begingroup$ thanks for the answer but I mean if I have three different models and I need to use them for a task and each one has a different number of tokens like in the question .. so is it a rule that the model which has large tokens means more efficient than the others .. that was my question $\endgroup$
    – Lei
    Jul 21, 2022 at 8:00
  • $\begingroup$ @Lei well as I said it's not so simple, there's definitely no such rule. It's possible that the larger model would perform better, but it's not sure. $\endgroup$
    – Erwan
    Jul 21, 2022 at 10:29
  • $\begingroup$ Appreciate your time and the answer .. Thanks a lot $\endgroup$
    – Lei
    Jul 21, 2022 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.