# Prediction Intervals Using XGBoost

I want to obtain the prediction intervals of my xgboost model which I am using to solve a regression problem. I am using the python code shared on this blog, and not really understanding how the quantile parameters affect the model (I am using the suggested parameter values on the blog). When I apply this code to my data, I obtain nonsense results, such as negative predictions for my target values while my target values are always over 10K. I don't understand how should this code vary according to my data and would really appreciate any help.

Differences in my data to the data that is used on the blog are:

• My distribution is Poisson like.
• I have over 100 features.

Note: I tried tuning the delta, threshold and var parameters, but they don't seem to have a controllable effect on the results and predictions remains nonsense.

To produce confidence intervals for xgboost model you should train several models (you can use bagging for this). Each model will produce a response for test sample - all responses will form a distribution from which you can easily compute confidence intervals using basic statistics. You should produce response distribution for each test sample.

• The OP is about prediction intervals, not c.i. Jun 10 '18 at 7:54

Try the following code. It must work fine. It might take a lot of time (more than 100 features).

Change max_depth to 6 if you want more accuracy. (because of 100 features.)

We can change learning_rate between 0 and 1, to improve the efficiency.

import xgboost as xgb
model_xgb = xgb.XGBRegressor(colsample_bytree=0.4603, gamma=0.0468,
learning_rate=0.05, max_depth=3,
min_child_weight=1.7817, n_estimators=4200,
reg_alpha=0.4640, reg_lambda=0.8571,
subsample=0.5213, silent=1,

X_train, X_test, Y_train, Y_test= train_test_split(X, Y, random_state= 0)
def model_score_error(model):
prepared_model=model.fit(X_train, Y_train)
x=prepared_model.score(X_test,Y_test)
print('Score: ',x)
Target_predicted=prepared_model.predict(X_test)
MSE=mean_squared_error(Y_test,Target_predicted)
print('mean square error', MSE)

model_score_error(model_xgb)

• thanks, but how this is calculating a prediction interval?
– mari
Dec 7 '17 at 16:37
• you might refer this : stats.stackexchange.com/questions/255783/… Dec 11 '17 at 19:49
• Many of these answers are for confidence intervals, not prediction intervals (which is a broader term). There's a very nice explanation of the difference between these two here- stats.stackexchange.com/a/16496/105753. May 28 '19 at 21:42