For example, if I run a single round (
nrounds=1), how does XGBoost go about making predictions? I thought it would simply return a linear regression model, but I quickly shot that theory down by working out an example.
gblinear uses linear functions, in contrast to
dart which use tree based functions.
One primary difference between linear functions and tree-based functions is the decision boundary. Tree-based models decision boundaries are only piece-wise, perpendicular rules to each feature. Linear functions are monotonic lines through the feature space.
Tree-based functions tend to be more flexible. If the data is distributed closer to a line, then a linear function will be more parsimonious.