1
$\begingroup$

I read the paper but found nothing talking about how to implement incremental learning.

Can someone share some basic or deep knowledge? not in coding way.

I know how to write code snippet to train incrementally.

When new data comes in, how to train incrementally if I use XGBRegressor? Reserve the old trees and train new data with new trees?

I found nothing talking about this in detail

$\endgroup$

1 Answer 1

3
$\begingroup$

In the fit method, parameter xgb_model can be specified to continue training an old model.

https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.XGBRegressor.fit

Edit: Yes, as I understand it, the old trees are preserved in this method. The new trees will fit the residuals of whatever data you pass: you could continue with only the new data, which will first be run through the existing trees and have their residuals fit by new trees; or continue with all the data, which will do the above but also use the old data when splitting (probably better, so that the model doesn't lose its fit to the old data, but maybe your use case cares about the new data substantially more?). In some cases, it may be better to just retrain from scratch (if the existing trees do a poor job on the new data).

$\endgroup$
9
  • 1
    $\begingroup$ I know, I am curious about the detail, how to determine the split point? or should save what in model? $\endgroup$ Mar 19, 2019 at 1:45
  • $\begingroup$ thx for updating. My data may too much to store in server, so I train new data incrementally and remove old data. After iterative training incrementally with new data, it predict result with the newer trees rather than old trees? $\endgroup$ Mar 19, 2019 at 5:07
  • 1
    $\begingroup$ No, after incremental training the model retains all the trees; any new predictions will get run through the entire set of trees. See also (1) stackoverflow.com/questions/38079853/… , (2) datascience.stackexchange.com/questions/25348/… , (3) github.com/dmlc/xgboost/issues/3055 $\endgroup$
    – Ben Reiniger
    Mar 19, 2019 at 11:50
  • $\begingroup$ It seems train Data_A incrementally -> train Data_B incrementally isn't equal to train (Data_A + Data_B) ? $\endgroup$ Mar 20, 2019 at 5:02
  • 1
    $\begingroup$ That's probably better as a new question. $\endgroup$
    – Ben Reiniger
    Mar 21, 2019 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.