For models trained with a moderate amount of data, I would expect a the development workflow to include extensive hyper-parameter optimization through extensive retraining.

But how is this handled for large models, like GPT-3? I read that it costs tens of thousands of dollar to train the model. I would assume that during development, a smaller amount of training data is used to optimize the hyper-parameters, but I assume what is good for less training data is not necessarily good for higher amounts of training data. For example, the more parameters a model has, the more it might(?) suffer from overfitting for smaller data sets, but might achieve better performance than a smaller model when more data is involved.

But if the training gets very expensive and time-intensive, how can experimentation still happen?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.