2
$\begingroup$

I am looking at a software development pipeline where I am predicting the lead time of different products flowing through the pipeline.

After applying a boxcox transformation on the lead time (target variable) and creating a XGBoost regressor model I can see that the residuals follow a t-locationScale distribution. enter image description here

So now I looked at this guide which describes a method to create a prediction interval for any regression model assuming that the residuals are normally distributed. https://qucit.com/a-simple-technique-to-estimate-prediction-intervals-for-any-regression-model_en/

But I tried to tweak it to my distribution.

So a t-locationScale distribution has a $\sigma$, $\mu$ and $\nu$ parameter. The variance is only defined for $\nu>2$. My specific distribution has $\nu = 2.56$ and $\mu = 0.04$, $\sigma = 0.97$ So I could take the 95% interval of this distribution and say that for any $\hat{y}$, the prediction interval is the 95% interval of the residual distribution.

But I want to take into consideration that the prediction interval should change with different inputs. I created a regressor model, which I trained and then made predictions using the validation set. I then took the square of the error and trained an additional error model on this data. Such that the error model could predict the variance of the residuals distribution.

  xgb = XGBoostRegressor()
  xgb.fit(X_train,y_train)
  y_hat = xgb.predict(X_val)
  val_error = (y_hat-y_val)**2

  xgb_error = XGBoostRegressor()
  xgb_error.fit(X_val, val_error)

  variance_hat_residuals = xgb_error.predict(X_test)

The relationship between variance and $\sigma$ and $\nu$ for a t-locationScale distribution is

var = $\sigma^2 *\frac{\nu}{\nu-2}$

Now here is where I make an assumption which I am not sure makes sense.

I assume that the degrees of freedom $\nu$ is the same as for all residuals, $\nu = 2.56$ and then I solve for $\sigma$ through the following.

$\hat{\sigma} = \sqrt{\frac{\hat{var}*(\nu-2)}{\nu}}$

And estimate the lower and upper quantiles from this distribution.

 residual_distribution_lower_quantile = scipy.stats.t.ppf(q = 0.025, df = 2.56, scale = sigma)
 residual_distribution_upper_quantile = scipy.stats.t.ppf(q = 0.0975, df = 2.56, scale = sigma)

I then predict the lead time $\hat{y}$ and say that the mean of the distribution is $\hat{y}$

   pred = xgb.prediction(X_test)
   lower_interval = pred + residual_distribution_lower_quantile
   upper_interval = pred + residual_distribution_upper_quantile

Does it make sense to make the claim of $\nu$ is static? My score for the prediction interval is now $81\%$ since I am clearly simplifying the problem.

Any suggestions for improving my method?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.