Up until now I have mostly been using neural nets for regression. There on each iteration (epoch) the weights are updated using the whole training data, and we expect the error to go down each time, at least on the training set (an up until some moment on the validation one).
I am now venturing into the XGBoost models for a change and I have noticed that its training progress is very much reminiscent of the one with NNs. Namely, instead of epochs here I have boost runs, and with each boost run I get better and better training results and up until some moment I expect the validation error to decrease as well.
What I'm confused about though is that XGBoost final prediction is the sum of all predictions made by each tree. A single tree is being trained every boost run. This is very much unlike the NN case, where after each epoch we kinda forget about the NN we had before and just focus on the one with the most recent weights.
It hence sounds to me that XGBoost model can be interpreted as an ensemble of predictors, where each predictor has prediction of some quality, and the final one is their combination. For example, the only boosting-like method I was familiar with before is the forward selection for linear regression. You initialize the error as the target, then on each step you pick the feature most correlated with the error, do a linear regression, substract prediction from the current error, get the new error etc. In such case the very first linear regression is the best single-target prediction, whereas the rest of the models are kinda correcting it for more refined cases. I thought that XGBoost would yield a similar behavior, but its first tree always gives me a very bad r2 score and it does look like anything "best among simple" that just needs to be corrected.
Is my intuition wrong and the first tree of XGBoost is very different from the first step of boosting in linear regression?
