I know it's a best practice to ship our code from dev to staging to production by including different level tests and validations that will help to confidently deploy on the production environment.

But, for the ML models, data scientists will first do experiments(train and validate) on the sample dataset and then use the full dataset to train and validate on finalized experiment config. If the performance is poor, then need to re-iterate by updating the sample dataset and start doing experiments.

My question is,

  1. Can we use production data while doing experiments and re-training on the best experiment?
  2. Do we train the model in three different environments? (IMO, No, but looking for justifications and understanding industry best practices)
  3. How do the dev, staging, and production environment work for model training in real-world scenarios?

ML models can be rule-based, recommendation models, or traditional ML/deep learning models.


2 Answers 2

  1. Yes - Production data should be used. The highest quality, newest data should be used to train a machine learning model. Typically, new data is used to fine-tune existing models.

  2. No - Training should be done in its own separate environment. A full-trained model then is moved into development, staging, and production environments for prediction-only (aka, inference). Training requirements are very different than prediction requirements. The goal of training is to find the best algorithm/architecture, hyperparameters, and parameters. The goal of prediction is to apply that solution to solve a user's problem. A prediction-only system will include many other features, including safeguards and fallback options.

  3. Not applicable, see see above.

  • $\begingroup$ That means, for training, there will be one single environment that will have the access to production data. I assume the infra configurations of this environment should be high. In this case, there will be a single model registry that contains the dev, staging, and production variant models. $\endgroup$ Feb 21 at 15:15
  • $\begingroup$ Yes, you are correct. $\endgroup$ Feb 21 at 15:45
  • $\begingroup$ Will this be applicable to rule-based and recommendation models? I have made slight changes to the question at the end. $\endgroup$ Feb 22 at 12:15
  • 1
    $\begingroup$ Applicable to all the models! Training is supposed to happen in dev (in your case) environment, else even better to do in a completely separate environment (ideal case) as Brian recommended. $\endgroup$ Feb 22 at 13:26
  1. Yes! You can take a dump of production data, merge with existing training data (with all processing steps) and retrain (as many number of experiments desired) your model. But before you do that, it would be advisable to test predictions on new (prod) data with existing in-prod model (eliminates hassle of retraining if the model is already generalizing well).

  2. All the 'experiments' (training/validation) should happen in dev environment. Staging is to ensure proper sync between model & new/incoming data for predictions before prod deployment.

  3. It is extremely important to keep the environment identical in all three stages (specially staging & prod). Variations in dev environment is still okay. But then, careful of versions (if applicable) during deployment in staging (before replicating in prod).

  • 2
    $\begingroup$ That means, the model will be trained once in the dev environment with production data and the same model pickle file/ artifacts will be moved to staging and production environemnt, right? $\endgroup$ Feb 21 at 11:22
  • $\begingroup$ @shaikmoeed Bingo! $\endgroup$ Feb 21 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.