Quick summary about the problem: we are trying to deploy our regression model, where the clients require "individual prediction error". Since we're predicting something unknown in advance, we can't measure the error the standard way of y_true - y_predicted.

I have done some research already, but the problems with existing methods are this:

  • Since we're using boosting algorithms (xgboost, catboost), we can't rely on normality assumptions and generate standard confidence intervals.

  • A solution proposed here is to train multiple models and get the average of it, but then it's not viable in the production level as it would become at least 3 times as slow to train / predict.

  • Another way is to create a quantile regression as stated here, but this would impact our accuracy, which we can't sacrifice.

  • Finally, we have tried to train a model using our validation set error and try to predict the test set error, but the accuracy is very very low.

So my question is this: Is there a way to estimate an individual prediction error, without knowing it's true value in advance, that wouldn't affect the model accuracy or speed?

  • $\begingroup$ What do you mean by individual prediction error? Is it like say my model predict $\hat{y}$ then you are supposed to output how "confident" it is? $\endgroup$ – Yohanes Alfredo Nov 20 '19 at 8:41
  • $\begingroup$ Yes, exactly that $\endgroup$ – Jurgis Samaitis Nov 20 '19 at 8:47
  • $\begingroup$ Is linear model enough? Have you read about bayesian linear regression? $\endgroup$ – Yohanes Alfredo Nov 20 '19 at 8:53
  • $\begingroup$ That's the problem - our main goal is to be as accurate as possible and linear models just aren't as accurate (although much more explainable). We haven't ran the bayesian regression though, I will definitely check it out. $\endgroup$ – Jurgis Samaitis Nov 20 '19 at 8:57
  • $\begingroup$ This is a good article towardsdatascience.com/… $\endgroup$ – Yohanes Alfredo Nov 20 '19 at 9:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.