XgBoost given targets its only feature but fails when test targets are outside the range of training targets?

Question

I'm learning to use XgBoost, and I'm doing an exercise involving predicting prices. However I'm noticing some weird behavior where XgBoost's predictions deviate from the target value even if I'm passing the target as the model's only feature.

This appears to happen when the test prices are significantly higher than the training set prices.

This is rather counter intuitive, as I'd imagine the model would always output the feature, but somehow it limits itself. Was wondering if anyone knows why this happens, and if there is someway around this.

params = {'gamma': 0.05, 'learning_rate': .05, 'max_depth': 8, 'n_estimators': 100, 'random_state': 10}

model = xgb.XGBRegressor(**params, objective='reg:squarederror')
model.fit(X_train, y_train, verbose=False)

y_pred = model.predict(X_test)

Answer 1

The output of the decision tree is the corresponding value on the leaf of the tree which is determined from the splits in the trees. The splits as well as the leaf values are parameters which are learnt. I recommend to watch this video which explains how a regression tree is trained and what it learns https://www.youtube.com/watch?v=g9c66TUylZ4

In summary, this means that a decision tree for regression (like the current XGBoost employs it) cannot perform well on data which follows a different distribution than the one seen in the training data set.

If you want to have some generalization beyond the training data you provided, consider trying out a linear tree, which is a tree where each leaf stands for a linear model. Checkout the python package https://github.com/cerlymarco/linear-tree which you can install via pip like this "pip install linear-tree".

Answer 2

It seems to follow expected behavior as there is no such values in the train set. The traditional time-series way to deal with this would be to try to predict increments instead of raw values. That way you would have all sort of increment in your train set. You would not avoid similar problem (how to predict large deviations if you don't have them in your train set). Please note that market values are usually very hard to deal with. you might want to check an introductory course of quantitative finance.