How to add confidence to model's prediction?

Question

I am a newbie in ML working on a time series prediction project. The objective is to predict the future outcome of a time series (int valued, with different upper bound, think of it as different sized parking lot availability prediction) based on its historical value.

I'm currently using a regression approach using slide windows algorithm. I tried different ML models and they seem to be working okay(better than my baseline at least).

Now I'm trying to add confidence to my prediction, something like "I have 95% confidence that the outcome would be 2". I'm thinking about using the prediction mean squared error as a metric.

The problem is,

1. Is it feasible to assume the prediction error follows gaussian distribution and add confidence based on that?
2. What distribution should I use for highly discrete state space? For example, when there is only 4 possible states {0,1,2,3}, and I predicted 2.5 with mse 1, how can I distribute the possibility over those states?

Any advice on the general model architecture and confidence set up will be appreciated!

Answer 1

It is common to say the error term follows a standard Guassian distribution. If you assume that to be true, then your squared errors follow a Chi-squared distribution:

In probability theory and statistics, the chi-squared distribution (also chi-square or χ2-distribution) with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables.

Have a look here for some ideas about how to implement a quasi-confidence metric, based on your (mean squared) errors. It assumes the errors follow a chi-aquared distribution and then uses the normalised RMSE to define a set of confidence boundaries for a given confidence level, $\alpha$, as follows:

$$ \left[\sqrt{\frac{n}{\chi_{1-\frac{\alpha}{2},n}^{2}}}RMSE,\sqrt{\frac{n}{\chi_{\frac{\alpha}{2},n}^{2}}}RMSE\right] $$

See the link for the steps involved. Here is the coded simulation taken from that post, with some added comments (requires python 3):

from scipy import stats
import numpy as np

s = 3                           # a constant to scale the random distribution
n = 4                           # number of samples/states per prediction
alpha = 0.05                    # confidence interval

# distribution with confidence intervals ɑ = 0.05
c1, c2 = stats.chi2.ppf([alpha/2, 1-alpha/2], n)

# we will take this many samples (this pre-allocates the y-vector)
y = np.zeros(50000)

# Loop over each sample and record the result mean sample
# This would be your prediction vector - here it is random noise
for i in range(len(y)):
    y[i] = np.sqrt(np.mean((np.random.randn(n)*s)**2))

# Use the chi-squared distributed confidence intervals to see when predictions fall
# finds percentage of samples that are inside the confidence interval
conf = mean((sqrt(n/c2)*y < s) & (sqrt(n/c1)*y > s))

print("1-alpha={:2f}".format(conf))

Here is another answer on CrossValidated, which gives more information around the area.

---

Additionally, if you assume your predictions lie within a Gaussian distribution, you could use the variance of your predictions as the confidence (welcome to Bayesian learning!).

There are packages that will help you do this, such as BayesOptimization. There are lots of examples on that webpage. Essentially, you will be able to make predictions and automatically get robust estimates for condifence... and some cool plots to show where your model is quite sure, and where it isn't:

Answer 2

Assuming Gaussian normal distribution is usual but sometimes unrealistic, especially if your model is a least square of identical independent observations. Even if it is rarely the case, many will propose very convincing arguments to justify the distribution of the forecast is normal, mainly because you can compute confidence intervals easily.

At side of these so called "exact test", you have resampling techniques, which are in facts the more practical way to answer your second question (discrete distribution). In two words, you create a simulated history with the same distribution as the observed history, make your forecast and repeat this a great number of time. Then you can make statistic on these simulated forecasts and measure (as opposed to compute) the confidence intervals.

The various resampling methods differ on how to make the the simulated sample. The two more common techniques are:

• The Jackknife: you forget one point, which makes $n$ simulated sample ($n$ being the size of the original sample).

• The Bootstrap: you take randomly $n$ points of the observed sample, some point being taken once, some point taken twice, some point taken 3 time, some points no taken at all. This will produce as many simulated sample with a similar distribution, and whose "average" is the observed sample.

I like the jackknife better because the bootstrap is problematic when the process evolves over time, but in your case, it seams you may use a bootstrap.