7
$\begingroup$

In a regression problem, is it possible to calculate a confidence/reliability score for a certain prediction given models like XGBoost or Neural Networks?

$\endgroup$
6
$\begingroup$

No matter the model, you can always use the non-parametric bootstrap to construct a confidence interval for any parameter, including predictions (which are actually random variables themselves but are reported as expectations). Here's the general procedure:

  1. Let $N$ denote the number of observations in your training data $X$, and $x_j$ denote the specific observation whose prediction, $\hat{y}_j$, you want a CI for.
  2. Let $K$ denote some number of resampling iterations (Must be $\ge 20$ for a CI with coverage $\ge 95\%$)
  3. For $i$ in $K$, draw a $N$ random samples from $X$ with replacement. Denote this $X_i^{*}$
  4. Train a model on $X_i^{*}$ and use this model to form a prediction on $x_j$. Call this $\hat{y}^{*}_{ji}$
  5. Estimate distributional parameters for $\hat{y}_j$ from your sample. A $100 - \alpha$ CI is given by the $\frac{\alpha}{2}$ and $100 - \frac{\alpha}{2}$ percentiles of $\hat{y}^{*}_{j}$.
$\endgroup$
  • 1
    $\begingroup$ Is there a way to get this confidence interval for a model that is already trained? $\endgroup$ – Rodrigo Nader Jan 10 '18 at 3:43
  • 1
    $\begingroup$ Not that I know of. If you pretend your residuals are i.i.d. (they're probably not with those models), you could estimate the distribution of the residuals directly and then derive prediction intervals from that. Not sure if that suits your need. If you're trying to identify which predictions your model is more or less "sure" about, this won't give you that. $\endgroup$ – David Marx Jan 10 '18 at 5:24
  • $\begingroup$ @davidmarx why do we need the iid assumption? if we have enough validation data, can't we look at the errors as the parameter to estimate, and build a second regression model for estimating them? $\endgroup$ – ihadanny Nov 25 '18 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.