In a regression problem, is it possible to calculate a confidence/reliability score for a certain prediction given models like XGBoost or Neural Networks?
$\begingroup$
$\endgroup$
asked Jan 6, 2018 at 18:36
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
No matter the model, you can always use the non-parametric bootstrap to construct a confidence interval for any parameter, including predictions (which are actually random variables themselves but are reported as expectations). Here's the general procedure:
- Let $N$ denote the number of observations in your training data $X$, and $x_j$ denote the specific observation whose prediction, $\hat{y}_j$, you want a CI for.
- Let $K$ denote some number of resampling iterations (Must be $\ge 20$ for a CI with coverage $\ge 95\%$)
- For $i$ in $K$, draw a $N$ random samples from $X$ with replacement. Denote this $X_i^{*}$
- Train a model on $X_i^{*}$ and use this model to form a prediction on $x_j$. Call this $\hat{y}^{*}_{ji}$
- Estimate distributional parameters for $\hat{y}_j$ from your sample. A $100 - \alpha$ CI is given by the $\frac{\alpha}{2}$ and $100 - \frac{\alpha}{2}$ percentiles of $\hat{y}^{*}_{j}$.
-
1$\begingroup$ Is there a way to get this confidence interval for a model that is already trained? $\endgroup$ Commented Jan 10, 2018 at 3:43
-
1$\begingroup$ Not that I know of. If you pretend your residuals are i.i.d. (they're probably not with those models), you could estimate the distribution of the residuals directly and then derive prediction intervals from that. Not sure if that suits your need. If you're trying to identify which predictions your model is more or less "sure" about, this won't give you that. $\endgroup$ Commented Jan 10, 2018 at 5:24
-
$\begingroup$ @davidmarx why do we need the iid assumption? if we have enough validation data, can't we look at the errors as the parameter to estimate, and build a second regression model for estimating them? $\endgroup$– ihadannyCommented Nov 25, 2018 at 17:50