I'm studying TensorFlow Extended and I can see that it's training pipeline includes a "Tuner" component for hyperparameter tuning. As a consequence, I'm wondering if inclusion of tuning is a good practice in case of a production pipeline (which, as in most cases, invoked iteratively from time to time with additional new training instances). I can see three possibilities:
- Separate hyperparameter-tuning experiment before production, and no tuning in production pipeline (this is what I used to use before)
- No intitial tuning-experiment but let TFX do it all the time (this seems supported by TFX)
- Somekind of a mixture: see some level of tuning in initial separate experiment but still perform tuning for other parameters in production
My problem with the 2nd approach is that hyperparameter-tuners are usually not fit for examination of all possible components (like number of layers, for instance), and anyways, some hyperparams tend to be stable in searches, so why search these in each training run?