2
$\begingroup$

I am a newbie in ML working on a time series prediction project. The objective is to predict the future outcome of a time series (int valued, with different upper bound, think of it as different sized parking lot availability prediction) based on its historical value.

I'm currently using a regression approach using slide windows algorithm. I tried different ML models and they seem to be working okay(better than my baseline at least).

Now I'm trying to add confidence to my prediction, something like "I have 95% confidence that the outcome would be 2". I'm thinking about using the prediction mean squared error as a metric.

The problem is,

  1. Is it feasible to assume the prediction error follows gaussian distribution and add confidence based on that?
  2. What distribution should I use for highly discrete state space? For example, when there is only 4 possible states {0,1,2,3}, and I predicted 2.5 with mse 1, how can I distribute the possibility over those states?

Any advice on the general model architecture and confidence set up will be appreciated!

$\endgroup$
2
$\begingroup$

It is common to say the error term follows a standard Guassian distribution. If you assume that to be true, then your squared errors follow a Chi-squared distribution:

In probability theory and statistics, the chi-squared distribution (also chi-square or χ2-distribution) with k degrees of freedom is the distribution of a sum of the squares of k independent standard normal random variables.

Have a look here for some ideas about how to implement a quasi-confidence metric, based on your (mean squared) errors. It assumes the errors follow a chi-aquared distribution and then uses the normalised RMSE to define a set of confidence boundaries for a given confidence level, $\alpha$, as follows:

$$ \left[\sqrt{\frac{n}{\chi_{1-\frac{\alpha}{2},n}^{2}}}RMSE,\sqrt{\frac{n}{\chi_{\frac{\alpha}{2},n}^{2}}}RMSE\right] $$

See the link for the steps involved. Here is the coded simulation taken from that post, with some added comments (requires python 3):

from scipy import stats
import numpy as np

s = 3                           # a constant to scale the random distribution
n = 4                           # number of samples/states per prediction
alpha = 0.05                    # confidence interval

# distribution with confidence intervals ɑ = 0.05
c1, c2 = stats.chi2.ppf([alpha/2, 1-alpha/2], n)

# we will take this many samples (this pre-allocates the y-vector)
y = np.zeros(50000)

# Loop over each sample and record the result mean sample
# This would be your prediction vector - here it is random noise
for i in range(len(y)):
    y[i] = np.sqrt(np.mean((np.random.randn(n)*s)**2))

# Use the chi-squared distributed confidence intervals to see when predictions fall
# finds percentage of samples that are inside the confidence interval
conf = mean((sqrt(n/c2)*y < s) & (sqrt(n/c1)*y > s))

print("1-alpha={:2f}".format(conf))

Here is another answer on CrossValidated, which gives more information around the area.


Additionally, if you assume your predictions lie within a Gaussian distribution, you could use the variance of your predictions as the confidence (welcome to Bayesian learning!).

There are packages that will help you do this, such as BayesOptimization. There are lots of examples on that webpage. Essentially, you will be able to make predictions and automatically get robust estimates for condifence... and some cool plots to show where your model is quite sure, and where it isn't:

BayesianOptimization example plot

| improve this answer | |
$\endgroup$
1
$\begingroup$

Assuming Gaussian normal distribution is usual but sometimes unrealistic, especially if your model is a least square of identical independent observations. Even if it is rarely the case, many will propose very convincing arguments to justify the distribution of the forecast is normal, mainly because you can compute confidence intervals easily.

At side of these so called "exact test", you have resampling techniques, which are in facts the more practical way to answer your second question (discrete distribution). In two words, you create a simulated history with the same distribution as the observed history, make your forecast and repeat this a great number of time. Then you can make statistic on these simulated forecasts and measure (as opposed to compute) the confidence intervals.

The various resampling methods differ on how to make the the simulated sample. The two more common techniques are:

  • The Jackknife: you forget one point, which makes $n$ simulated sample ($n$ being the size of the original sample).

  • The Bootstrap: you take randomly $n$ points of the observed sample, some point being taken once, some point taken twice, some point taken 3 time, some points no taken at all. This will produce as many simulated sample with a similar distribution, and whose "average" is the observed sample.

I like the jackknife better because the bootstrap is problematic when the process evolves over time, but in your case, it seams you may use a bootstrap.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your reply. I'm a bit confused by the resampling part tho. In my case, I have known training set X, which is generated from a known time series through a sliding-window algorithm. The mean-squared error is estimated between the model's prediction and training target y, which is the real value of future observation. In that case, how can I resample my training set? $\endgroup$ – plpopk Aug 14 '18 at 14:22
  • $\begingroup$ Short answer: put the n values of your training set in a hat, and take one out. Repeat this n times replacing the value before the draw. This gives you a simulated sample which has the same size and the statistical properties as the original. Compute the forecast on it. Repeat this 1600 times. This gives you 1600 different values for the forecast. Sort them, take the 80-th (5%) and the 1520-th (95%). This gives you a confidence interval which is (statistically) the same as the confidence interval on your forecast. Real fun to code. Greetings goes to Efron-1979. $\endgroup$ – AlainD Aug 14 '18 at 15:10
  • $\begingroup$ Alternative answer (bootstrap on the training set): split your data randomly in a training set and a validation set. Make your prediction. Repeat this 1600 times. Then the confidence intervals (obtained by dropping the first and last 80 forecast) is (statistically) the confidence interval of your forecast. Note (while no academic specialist is looking around), if you put the right wording and conditions, this method is called cross-validation. $\endgroup$ – AlainD Aug 14 '18 at 15:10
  • $\begingroup$ By while no academic specialist is looking I mean: for the boostrap you measure the error on the whole data, while for cross-validation you measure the error on the validation sample. The error is of course bigger with the cross-validation than with the bootstrap. But that's for the error, and you are not after a confidence interval for the error. The parameters (the coefficients of the regression) will be the same, as well the forecast value. And you are looking for a confidence on the forecast, aren't you? $\endgroup$ – AlainD Aug 15 '18 at 8:49
  • $\begingroup$ Hi, I understand the bootstrapping part. It's one of the standard way to do cross-validation for normal machine learning training set. Actually, the RMSE I used for creating confidence interval is calculated on my CV set. However, to me it seems resampling on input data can show the uncertainty on prediction caused by training set, while my problem is how to show the intrinsic between model prediction and future real values. It always exist since my time series have a lot of random variation. $\endgroup$ – plpopk Aug 15 '18 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.