Is it a good practice to use hyperparameter tuning in production pipeline?

Question

I'm studying TensorFlow Extended and I can see that it's training pipeline includes a "Tuner" component for hyperparameter tuning. As a consequence, I'm wondering if inclusion of tuning is a good practice in case of a production pipeline (which, as in most cases, invoked iteratively from time to time with additional new training instances). I can see three possibilities:

• Separate hyperparameter-tuning experiment before production, and no tuning in production pipeline (this is what I used to use before)
• No intitial tuning-experiment but let TFX do it all the time (this seems supported by TFX)
• Somekind of a mixture: see some level of tuning in initial separate experiment but still perform tuning for other parameters in production

My problem with the 2nd approach is that hyperparameter-tuners are usually not fit for examination of all possible components (like number of layers, for instance), and anyways, some hyperparams tend to be stable in searches, so why search these in each training run?

Answer 1

Hyperparameter tuning helps model to find best parameters so that it can give you optimal results. Hyperparameter tuning should be done during training process only and not in production pipeline if you are deploying the model.

But if your doing model retraining as part of production pipeline then hyperparameter tuning should be done to get best results.

Your second approach is about how to optimise Hyperparameters tuning as it is very computation intensive task.

1. Stepwise Hyperparameter tuning to reduce the number of iteration required for example for randomforest first find the optimal depth and fix it in next step.

2. Instead of grid Search use a smart hyperparam tuning like optuna ,hyperopt to reduce the time taken.

3. If you know some of the best hyperparam and know they are stable you dont need to tune those.