2
$\begingroup$

Every example I come across any kind of iterative learning on Random Forest/XGBoost/LightGBM, it just continuously grows the number of estimators for new batches of data by n_tree/n_estimators/num_boost_rounds every time .fit() gets applied [...]. Most of them seem to rely on iterative learning for training on very large datasets that can't be loaded into memory at once.

However, I want to implement a continuous learning pipeline (with LightGBM; Python) that takes newly available data on a daily basis in order to update an existing model (without the need to retrain on the whole [growing] dataset; stateful). The initially mentioned approach would imply that my model will ever increase its tree count.

Is it possible to train tree-based algorithms so that the estimators (split thresholds) themselves get updated/adjusted in contrast to only adding estimators?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
3
$\begingroup$

This is a really good question for which I will give you a theoretical result; in particular, I am not aware of any specific implementation in any programming language.

The concept of incremental learning with decision trees started in 1986 to enhance the ID3 learning algorithm to learn continually/incrementally (recall that ID3 deals only with categorical input features/variables); the resulting procedure is called ID4.

Some years later, Utgoff et al. proposed other two approaches (ID5 and ITI) to overcome some of the shortcomings of the ID4 approach.

There is also a nice Wikipedia page that you might read.

EDIT

There are also works on online boosting and bagging, which I was not aware of either, but your question thought me something new; thanks for that (+1)!

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Highly appreciate your feedback! Your answer also reminded me of the concept of 'Hoeffding [Decision] Trees' as predecessor of EFDT ('Extremely Fast Decision Tree'; as mentioned in the linked Wikipedia article) - although they assume a static data distribution over time. An implementation of Hoeffding Trees for Regreesion can be found in scikit-multiflow (scikit-multiflow.readthedocs.io/en/stable/index.html) for stream processing. I have to wrap my head around, whether this is a viable approach for my particular problem. I'll leave the question open for now - thank you! $\endgroup$
    – OliverHennhoefer
    Commented Sep 13, 2022 at 14:11
  • 2
    $\begingroup$ Yes, you are right, I also wanted to say something about those two. I just wanted to keep things as simple as possible as both research papers are very technical; although, are very good reading. I appreciate that there are implementations, and this is an opportunity for my research team and I to enhance such implementations with different degrees of expressivity (+1 to your comment). $\endgroup$
    – Eduard
    Commented Sep 13, 2022 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.