# Prediction error without having a true value

Quick summary about the problem: we are trying to deploy our regression model, where the clients require "individual prediction error". Since we're predicting something unknown in advance, we can't measure the error the standard way of y_true - y_predicted.

I have done some research already, but the problems with existing methods are this:

• Since we're using boosting algorithms (xgboost, catboost), we can't rely on normality assumptions and generate standard confidence intervals.

• A solution proposed here is to train multiple models and get the average of it, but then it's not viable in the production level as it would become at least 3 times as slow to train / predict.

• Another way is to create a quantile regression as stated here, but this would impact our accuracy, which we can't sacrifice.

• Finally, we have tried to train a model using our validation set error and try to predict the test set error, but the accuracy is very very low.

So my question is this: Is there a way to estimate an individual prediction error, without knowing it's true value in advance, that wouldn't affect the model accuracy or speed?

• What do you mean by individual prediction error? Is it like say my model predict $\hat{y}$ then you are supposed to output how "confident" it is? – Yohanes Alfredo Nov 20 at 8:41
• Yes, exactly that – Jurgis Samaitis Nov 20 at 8:47
• Is linear model enough? Have you read about bayesian linear regression? – Yohanes Alfredo Nov 20 at 8:53
• That's the problem - our main goal is to be as accurate as possible and linear models just aren't as accurate (although much more explainable). We haven't ran the bayesian regression though, I will definitely check it out. – Jurgis Samaitis Nov 20 at 8:57
• This is a good article towardsdatascience.com/… – Yohanes Alfredo Nov 20 at 9:24