For models trained with a moderate amount of data, I would expect a the development workflow to include extensive hyper-parameter optimization through extensive retraining.
But how is this handled for large models, like GPT-3? I read that it costs tens of thousands of dollar to train the model. I would assume that during development, a smaller amount of training data is used to optimize the hyper-parameters, but I assume what is good for less training data is not necessarily good for higher amounts of training data. For example, the more parameters a model has, the more it might(?) suffer from overfitting for smaller data sets, but might achieve better performance than a smaller model when more data is involved.
But if the training gets very expensive and time-intensive, how can experimentation still happen?