0
$\begingroup$

So I understand the intuition after reading and watching many of Tianqi Chen and Tong He's papers and talks. But in reality, if you have a dataset, how do you fit another classification tree based on the residuals of the tree from the last round?

Say I have the following iris dataset:

enter image description here

And I get 3 error after running a classification tree. Then how do I fit another classification tree based on what I got wrong?

And after then, how would I weight them to form an ensemble?

$\endgroup$
3
  • $\begingroup$ This post (medium.com/mlreview/gradient-boosting-from-scratch-1e317ae4587d) for example helped me a lot to understand gradient boosting trees. Weighting is not used in this example. But don't you think they simply fit another tree on residuals (it might be easier to think of regression) and add that to previous tree as it is like this example (or as in reality a weighted addition of predictions on residual depending on how much we want to focus on mistakes)? Whether what is shown here is exactly what happens in XGboost, still remains a mystery! $\endgroup$ – TwinPenguins Feb 13 '18 at 8:24
  • $\begingroup$ Right, I've seen this post as well, but calculating the residuals in a logistic regression or a linear regression as in the post is straight forward, but how do you actually calculate the residuals of a tree? Do you just take all your wrongly predicted results and form a root node from there on? $\endgroup$ – stanley4430 Feb 13 '18 at 17:01
  • $\begingroup$ OK. I am losing you here. Again to picture it easier, in a simple tree for regression, once a tree is fitted, residuals are a matter of finding the standard error (deviations from the tree predictions at each point). Then from there you fit another tree to that residuals as you did in the previous step and so on. I did not understand why "logistic" or " linear regression" were used in that blog post. An actual tree was modelled and there is a link to that Class Decision Tree he uses. Maybe you need to read the blog more carefully. $\endgroup$ – TwinPenguins Feb 14 '18 at 13:11
2
$\begingroup$

There are 2 methods to "continue a model training":

Training from another xgboost model

If you have a pre-trained xgboost model, you can continue training using the xgb_model parameter in xgb.train.

xgb_model a previously built model to continue the training from. Could be either an object of class xgb.Booster, or its raw data, or the name of a file with a previously saved model.

Training residual

Using xgboost to training a residual or prediction of a model, you can following the demo provided in xgboost: https://github.com/dmlc/xgboost/blob/master/R-package/demo/boost_from_prediction.R

Simply speaking, you can use the "base_margin" element when preparing data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.