Does it make sense to calculate prediction interval for very small dataset (about 60 samples)? I know it's easy for linear regression. But it seems like for tree-based methods or non-linear methods it doesn't make any sense. Am I wrong?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ What would the prediction interval be for a tree-based model anyway? Regardless of data size, there's no standard definition that I'm aware of. $\endgroup$– shadowtalkerOct 8, 2018 at 15:22
-
$\begingroup$ You can use Quantile Regression Forests, for example, to calculate prediction interval $\endgroup$– DmytroSytroOct 8, 2018 at 15:49
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
In the case of quantile modeling, it makes sense as long as the quantile model makes sense. If you are able to build a good quantile model with 60 observations, then your estimates of intervals between quantiles should also be good. Keep in mind how errors will propagate when building intervals:
- The bias in the width of the interval will be the sum of the biases in the quantile estimates; they might balance out, or they might make things worse.
- The variance of the width of the interval will also be the sum of the variances of the quantile estimates. This could be a pretty big number, knowing that with 60 observations the "true" model variance will be large and hard to estimate.
Outside of that, it all depends on the model. Common statistical models tend to have explicit expressions for prediction intervals. In MCMC-based inference they can be inferred from the sampled values. In these cases, 60 is as good as 6,000 as long as the model works. As above, keep in mind that error propagation is not always easy to control. If these intervals are needed for serious applications, it's a good idea to generate fake data and run a sensitivity analysis.
The most generic solution is to obtaining a prediction interval is resampling. With just 60 observations, you can run full Leave-One-Out cross-validation (LOOCV). Or you can do bootstrap resampling. It might be a good idea, or the variance of the procedure will be through the roof. Again, simulation-based sensitivity analysis will help you decide.
-
$\begingroup$ How can I obtain prediction interval with leave-one-out? As I understand, I have 60 models and 60 predictions. And how can fake data help in sensitivity analysis? Thank you! $\endgroup$ Oct 8, 2018 at 19:54
-
1$\begingroup$ @DmytroSytro any resampling (bootstrap, CV, LOOCV) gives you a distribution of models, and you use that to estimate the prediction interval. Just make sure you use a nested loop if you are doing hyperparameter selection. $\endgroup$ Oct 8, 2018 at 20:25
-
1$\begingroup$ @DmytroSytro in sensitivity analysis, you would simulate a large "population", draw 60 sample records from it, and fit your model. The objective is to see how the model changes in response to different underlying assumptions. $\endgroup$ Oct 8, 2018 at 20:26