My question is about automating the training of the model as more data becomes available.

In this scenario, I have 1MM items that I split into training and test datasets to train, validate and ultimately deploy the model.

As more data becomes available on a daily basis, is it common in real-life projects or even advisable to automate the process above, as to leverage the new data and potentially make the model more accurate?


1 Answer 1


This is a big topic with a lot of possible solutions depending on your context, I'll just provide some of my personal experience from projects I've built.

I would say that it is common in industry to retrain models in an ongoing fashion as new data comes in. Many times this happens at the daily level using either a basic cron-like solution or an enterprise orchestration solution (e.g. Airflow/Kubeflow). Cron is easy to set up and manage for small/personal projects, the others require more engineering set up to manage but they also are more powerful/fully featured.

Once you have a model pipeline which trains in an ongoing basis, you would then need to decide whether to promote the new model to production or keep your existing model running. This could be done automatically given certain conditions (e.g. accuracy of new model > old model) or could require a manual step to evaluate the new model and promote after review. Many times the answer comes down to data quality/stability. For instance, if the input data has intermittent quality issues, you would be less likely to retrain and promote automatically without manual oversight.

Potentially helpful links:

Cron for ML: https://towardsdatascience.com/why-crontab-is-the-ultimate-data-science-back-end-tool-e3f212f2b13d

Airflow for ML: https://towardsdatascience.com/10-minutes-to-building-a-machine-learning-pipeline-with-apache-airflow-53cd09268977

Kubeflow Pipelines: https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/

  • $\begingroup$ Thank you for sharing your experience and those links!! $\endgroup$ Sep 13, 2020 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.