0
$\begingroup$

For models trained with a moderate amount of data, I would expect a the development workflow to include extensive hyper-parameter optimization through extensive retraining.

But how is this handled for large models, like GPT-3? I read that it costs tens of thousands of dollar to train the model. I would assume that during development, a smaller amount of training data is used to optimize the hyper-parameters, but I assume what is good for less training data is not necessarily good for higher amounts of training data. For example, the more parameters a model has, the more it might(?) suffer from overfitting for smaller data sets, but might achieve better performance than a smaller model when more data is involved.

But if the training gets very expensive and time-intensive, how can experimentation still happen?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.