In the project I am currently working on (predicting whether or not someone will click on some item from the mailing list that I send), each day data about users is extracted and the models are learned from scratch. Since users from yesterday don't really differ that much from users today I decided to try to somehow use the old models.
For the Deep Net and SVM models it's easy - I just make the algorithms start with the weights that were obtained last time and make them work through the new dataset using gradient descent.
What about the XGBoost though? I see that I have two options to use previous data:
The first one looks like it will make the new model focus too much on the old results, whereas the second one also feels a bit weird: I built 1000 trees and now I boost them with new data - so here it looks like I will end up mostly disregarding the old data and just rebuilding the forest to match the new data.
My first thought was to simply use half of trees from the old model and half from the new one. This way I would still use old knowledge but balance it out with new data. After some time I would get something like a funny pyramid-shaped forest with very few "old" trees and very many "young" ones (another way to think about it is like a power series model where data from period t-1 is given half the importance that data from period t gets). The idea looks neat, but I haven't seen anyone do that anywhere, so I remain skeptical.
Are there any options that I missed? I would be grateful for both comments on the possibilities that I outlined above and new ideas on treating this problem.