Do model training pipeline should run on dev, staging and production environment?

Question

I know it's a best practice to ship our code from dev to staging to production by including different level tests and validations that will help to confidently deploy on the production environment.

But, for the ML models, data scientists will first do experiments(train and validate) on the sample dataset and then use the full dataset to train and validate on finalized experiment config. If the performance is poor, then need to re-iterate by updating the sample dataset and start doing experiments.

My question is,

1. Can we use production data while doing experiments and re-training on the best experiment?
2. Do we train the model in three different environments? (IMO, No, but looking for justifications and understanding industry best practices)
3. How do the dev, staging, and production environment work for model training in real-world scenarios?

Note:

ML models can be rule-based, recommendation models, or traditional ML/deep learning models.

Answer 1

1. Yes - Production data should be used. The highest quality, newest data should be used to train a machine learning model. Typically, new data is used to fine-tune existing models.

2. No - Training should be done in its own separate environment. A full-trained model then is moved into development, staging, and production environments for prediction-only (aka, inference). Training requirements are very different than prediction requirements. The goal of training is to find the best algorithm/architecture, hyperparameters, and parameters. The goal of prediction is to apply that solution to solve a user's problem. A prediction-only system will include many other features, including safeguards and fallback options.

3. Not applicable, see see above.

Answer 2

1. Yes! You can take a dump of production data, merge with existing training data (with all processing steps) and retrain (as many number of experiments desired) your model. But before you do that, it would be advisable to test predictions on new (prod) data with existing in-prod model (eliminates hassle of retraining if the model is already generalizing well).

2. All the 'experiments' (training/validation) should happen in dev environment. Staging is to ensure proper sync between model & new/incoming data for predictions before prod deployment.

3. It is extremely important to keep the environment identical in all three stages (specially staging & prod). Variations in dev environment is still okay. But then, careful of versions (if applicable) during deployment in staging (before replicating in prod).