0
$\begingroup$

I have been working on a predictive model. With each prediction, we need to provide a score to express the confidence about our prediction. So I am looking at prediction interval (PI). In linear regression, I believe these can be obtained and well-documented. However, I am yet to find much reference for non-linear regression (such svr, gbr or other blackbox method for regression). Two methods that I have seen are given below:

1) Using bagging, we can generate many point prediction of each new data point, and then we get the interval from the distribution of these predictions around each new point.

2) using Quantile regression to get the upper and lower bound of the new point.

Personally, I do like the bagging method, although I don't feel very convinced. Hence, I am reaching out to the community to get a general opinion or some other ideas that I haven't seen so far.

Improve this question
$\endgroup$
2
$\begingroup$

Bootstrapping (which I think you are referring to as bagging, a specific algorithm that incorporates bootstrapping) is your friend here.

Simply train N models using N datasets, where these N datasets are created by sampling from your original dataset with replacement. Using these N different models, generate N predictions on an observation of interest.

You can now use any sort of method to derive the confidence interval of interest with these N predictions, whether that is using a normality assumption, empirical percentiles (simple bootstrapping), etc.

The one drawback I see with this method is that if you have many observations to predict, this will quickly become computationally expensive. That being said, bootstrapping is easily made parallel.

Improve this answer
$\endgroup$
  • $\begingroup$ One reason for my uneasiness with this method is that each time you are generating essentially a new model (different from the other models). Having said that, the only answer I can think off right now is that since we are sampling with replacement from the same dataset, models are essentially similar, providing us slightly different values, which account for the inherent uncertainty in prediction. Does that sound right? \ $\endgroup$ – user62198 May 3 '19 at 16:00
  • 1
    $\begingroup$ The whole point of bootstrapping in this context is to generate slightly different models with slightly different predictions. Otherwise, how can we get the desired measure of uncertainty if all of the models produced are the same? Having slightly different values is exactly what we want because like you say, we want to capture the uncertainty in our prediction. Even if the models are completely different, this is still not inherently a problem; your prediction intervals will just be incredibly wide which is informative as well (just means there is large uncertainty in your prediction). $\endgroup$ – aranglol May 3 '19 at 17:34
  • 1
    $\begingroup$ Ideally, we would like to use a closed form expression to generate our prediction intervals, but these require that errors follow a known probability distribution (so that we can make statements about our confidence). We don't know the distribution of our prediction errors but bootstrapping allows us to estimate these quantities empirically without any such assumption. $\endgroup$ – aranglol May 3 '19 at 17:41
  • 1
    $\begingroup$ So in the end, yeah I agree with you in that bootstrapping allows us to derive estimates for the uncertainty in our predictions (in this context). But, the similarity or dissimilarity in the models created (and how much their predictions vary) through the re-sampling procedure is in essence exactly what we what to estimate. $\endgroup$ – aranglol May 3 '19 at 17:44
1
$\begingroup$

The two approaches you mention are both valid.

What I would do is to check first the possibility of having confidence intervals for the $\beta$s and apply them to your data (just as you would do in a linear regression) but I am not sure if all the linear regression models have these confidence intervals.

After discarding this possibility, your first choice should be Bagging, mainly because with Quantile Regression you are obtaining a totally different model.

The only setback I see in Bagging is that I don't see any way to repeat the process quickly and obtaining many models just for one point data, but after solving this you should bag over the models.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.