I know it's a best practice to ship our code from dev to staging to production by including different level tests and validations that will help to confidently deploy on the production environment.
But, for the ML models, data scientists will first do experiments(train and validate) on the sample dataset and then use the full dataset to train and validate on finalized experiment config. If the performance is poor, then need to re-iterate by updating the sample dataset and start doing experiments.
My question is,
- Can we use production data while doing experiments and re-training on the best experiment?
- Do we train the model in three different environments? (IMO, No, but looking for justifications and understanding industry best practices)
- How do the dev, staging, and production environment work for model training in real-world scenarios?
ML models can be rule-based, recommendation models, or traditional ML/deep learning models.