I have trained and saved a data processing pipeline and an LGBM regressor on 3 months of historical data. Now I know that I can retrain the LGBM regressor on new data every day by passing my trained model as init_model for .train function. How do I retrain my sklearn pipeline that does the data processing using this new data?

One way I can think of is to monitor the feature drift and retrain pipeline for latest 3 months data when it crosses a certain threshold. Is there any better way to do this that I might be missing?

  • $\begingroup$ Why dont just retrain the LGBM and not the whole pipeline? Many of the transformers don't have a retrain option $\endgroup$ Commented Feb 4, 2022 at 13:12
  • $\begingroup$ @CarlosMougan yes, I can train just the LGBM but the transformers processing the data are then not adapting to the shift in feature distributions over time. Are you saying that is not likely to cause any issues in the future? $\endgroup$ Commented Feb 4, 2022 at 14:03
  • $\begingroup$ Why dont you just retrain the whole thing? $\endgroup$ Commented Feb 4, 2022 at 15:57


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.