Prediction Intervals Using XGBoost

I want to obtain the prediction intervals of my xgboost model which I am using to solve a regression problem. I am using the python code shared on this blog, and not really understanding how the quantile parameters affect the model (I am using the suggested parameter values on the blog). When I apply this code to my data, I obtain nonsense results, such as negative predictions for my target values while my target values are always over 10K. I don't understand how should this code vary according to my data and would really appreciate any help.

Differences in my data to the data that is used on the blog are:

• My distribution is Poisson like.
• I have over 100 features.

Note: I tried tuning the delta, threshold and var parameters, but they don't seem to have a controllable effect on the results and predictions remains nonsense.

To produce confidence intervals for xgboost model you should train several models (you can use bagging for this). Each model will produce a response for test sample - all responses will form a distribution from which you can easily compute confidence intervals using basic statistics. You should produce response distribution for each test sample.

• The OP is about prediction intervals, not c.i. Jun 10, 2018 at 7:54

Try the following code. It must work fine. It might take a lot of time (more than 100 features).

Change max_depth to 6 if you want more accuracy. (because of 100 features.)

We can change learning_rate between 0 and 1, to improve the efficiency.

import xgboost as xgb
model_xgb = xgb.XGBRegressor(colsample_bytree=0.4603, gamma=0.0468,
learning_rate=0.05, max_depth=3,
min_child_weight=1.7817, n_estimators=4200,
reg_alpha=0.4640, reg_lambda=0.8571,
subsample=0.5213, silent=1,

X_train, X_test, Y_train, Y_test= train_test_split(X, Y, random_state= 0)
def model_score_error(model):
prepared_model=model.fit(X_train, Y_train)
x=prepared_model.score(X_test,Y_test)
print('Score: ',x)
Target_predicted=prepared_model.predict(X_test)
MSE=mean_squared_error(Y_test,Target_predicted)
print('mean square error', MSE)

model_score_error(model_xgb)

• thanks, but how this is calculating a prediction interval?
– mari
Dec 7, 2017 at 16:37
• you might refer this : stats.stackexchange.com/questions/255783/… Dec 11, 2017 at 19:49
• Many of these answers are for confidence intervals, not prediction intervals (which is a broader term). There's a very nice explanation of the difference between these two here- stats.stackexchange.com/a/16496/105753. May 28, 2019 at 21:42

As an alternative to Nidhi's answer, you could also train two models to directly predict the bounds of the confidence interval. One model would have the upper bound as its training target, the other model the lower bound.

One downside of this approach is that you cannot simply change the confidence interval, because that requires retraining of the models.

• Maybe please describe how to do that? E.g. using quantile regression like here towardsdatascience.com/… ? Oct 31, 2022 at 9:13