# How to get a confidence score for predictions?

In a regression problem, is it possible to calculate a confidence/reliability score for a certain prediction given models like XGBoost or Neural Networks?

1. Let $N$ denote the number of observations in your training data $X$, and $x_j$ denote the specific observation whose prediction, $\hat{y}_j$, you want a CI for.
2. Let $K$ denote some number of resampling iterations (Must be $\ge 20$ for a CI with coverage $\ge 95\%$)
3. For $i$ in $K$, draw a $N$ random samples from $X$ with replacement. Denote this $X_i^{*}$
4. Train a model on $X_i^{*}$ and use this model to form a prediction on $x_j$. Call this $\hat{y}^{*}_{ji}$
5. Estimate distributional parameters for $\hat{y}_j$ from your sample. A $100 - \alpha$ CI is given by the $\frac{\alpha}{2}$ and $100 - \frac{\alpha}{2}$ percentiles of $\hat{y}^{*}_{j}$.