0
$\begingroup$

I've created a model for out-of-sample forecasting that uses multistep recursive strategy to reduce my problem to regression, the predictions are sufficient but I was wondering if there is any possibility to add something like prediction intervals - I know how to do this using residual based solutions (e.g. RMSFE) only for in-sample models when historical data is available.

I've read somewhere about an idea of using Bayesian Structural Time Series modelling on past forecast errors to somehow estimate the uncertainty and then Monte Carlo Simulation to generate a range of possible future errors but I've concerns about this solution.

$\endgroup$
1
  • $\begingroup$ Welcome to DS stack exchange. Please provide further info and include some references. Isn't Bayesian Structural Time Series model for Nowcasting and Predicting the present? How do you want to use it for Forecasting? Is it capable to do that if you know? $\endgroup$
    – Mario
    Oct 2, 2023 at 11:57

1 Answer 1

0
$\begingroup$

We can build a Prediction Intervals (PIs) once we have a set of predictions.

  • Method 1: RMSFE (Root Mean Squared Forecasting Error)

RMSFE is similar to RMSE. The only difference is that RMSFE has to be calculated on residual terms from predictions over unseen data (ie. Validation-set or Test-set).

  • Method 2: BCVR (Bootstrapping Cross-Validation Residuals) which is a theoretical method proposed by Brendan Artley in his online article:

    • Time Series Forecasting: Prediction Intervals Estimate the range of a future observation with confidence

Note: BCVR method is a theoretical proposal, it's not official yet. Please double-check if you wanna use it in scientific publications. I have not found an official paper for that. At the moment, there is a pre-print paper under the title of "Bootstrapping the Cross-Validation Estimate" from Bryan Cai et. al


All I am familiar with are methods based on residuals of past timestamps (e.g., RMSFE). Is it possible to somehow incorporate those into future timestamps?

Yes, this can be achieved using the Recursive multi-step forecasting strategy depicted in the following: img

You can deploy package and use it with the classes ForecasterAutoreg & ForecasterAutoregCustom based on the docs

Also, you can set regressor = lightgbm.LGBMRegressor() from docs if you wish.

I achieved these results using RMSE; you can change and use the abovementioned methods. You can extend it:

  1. for RMSFE by:

    RMSFE = np.sqrt(sum([x**2 for x in residuals]) / len(residuals))
    band_size = 1.96*RMSFE
    
  2. for BCVR, check the proposed author's implementation

img

You can reproduce what I achieved it using this example; otherwise, let me know.

$\endgroup$
6
  • $\begingroup$ I'm not sure if you understood my question correctly, I'm familiar with method that you provided but what if there isn't any known target values for test set, actually there is not even a test set itself but only timestamps from future with no relevant information. I train my model and forecast using multistep recursive strategy but don't know how to create prediction intervals. Maybe some residuals estimation from past values is worth trying? $\endgroup$
    – kkkk0
    Oct 2, 2023 at 11:31
  • $\begingroup$ Then, You need to frame\edit your question better about what you are going to achieve. I stated your question exactly in my answer before answering. you might look for regression models and do out-of-sample prediction (predicting beyond the training dataset) over future data (unseen test-set) despite your strategy (e.g. residuals estimation) $\endgroup$
    – Mario
    Oct 2, 2023 at 11:47
  • $\begingroup$ Are you looking for PIs that a model generates forecasts for the next $n$ days, only the ($n$-day-ahead forecast) like this? $\endgroup$
    – Mario
    Oct 2, 2023 at 12:07
  • $\begingroup$ Yes, that looks like something I want to achieve $\endgroup$
    – kkkk0
    Oct 2, 2023 at 12:33
  • $\begingroup$ Can you please provide me from where did you get that or any other source that will help me achieve this type of results? $\endgroup$
    – kkkk0
    Oct 2, 2023 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.