5
$\begingroup$

In the project I am currently working on (predicting whether or not someone will click on some item from the mailing list that I send), each day data about users is extracted and the models are learned from scratch. Since users from yesterday don't really differ that much from users today I decided to try to somehow use the old models.

For the Deep Net and SVM models it's easy - I just make the algorithms start with the weights that were obtained last time and make them work through the new dataset using gradient descent.

What about the XGBoost though? I see that I have two options to use previous data:

  1. Make predictions using the old model and feed them as a parameter to the new model

  2. Use the old booster

The first one looks like it will make the new model focus too much on the old results, whereas the second one also feels a bit weird: I built 1000 trees and now I boost them with new data - so here it looks like I will end up mostly disregarding the old data and just rebuilding the forest to match the new data.

My first thought was to simply use half of trees from the old model and half from the new one. This way I would still use old knowledge but balance it out with new data. After some time I would get something like a funny pyramid-shaped forest with very few "old" trees and very many "young" ones (another way to think about it is like a power series model where data from period t-1 is given half the importance that data from period t gets). The idea looks neat, but I haven't seen anyone do that anywhere, so I remain skeptical.

Are there any options that I missed? I would be grateful for both comments on the possibilities that I outlined above and new ideas on treating this problem.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Your class of problems is called data stream mining in the literature. If you google data stream mining and gradient boosting, you'll find plenty of stuff. $\endgroup$
    – Ricardo Magalhães Cruz
    Feb 3, 2017 at 14:20
  • $\begingroup$ Since there is a lot that you need to understand, you can go through the following online tutorial. Its a webpage, explaining about xgboost from the scratch. Hopefully, it will clarify your doubts from the beginning. machinelearningmastery.com/… $\endgroup$
    – Nidhi Agarwal
    Dec 6, 2017 at 17:10

1 Answer 1

Reset to default
1
$\begingroup$

Why not just keep training your old model? just pass the previous xgb model (path + filename) as an argument in the train function.

bst = xgb.train(param0, dtrain, num_round, evals=[(dtrain, "training")], xgb_model=prev_xgb_model)

xgb_model=prev_xgb_model
Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.