Does it make sense to calculate prediction interval for very small dataset (about 60 samples)? I know it's easy for linear regression. But it seems like for tree-based methods or non-linear methods it doesn't make any sense. Am I wrong?
In the case of quantile modeling, it makes sense as long as the quantile model makes sense. If you are able to build a good quantile model with 60 observations, then your estimates of intervals between quantiles should also be good. Keep in mind how errors will propagate when building intervals:
- The bias in the width of the interval will be the sum of the biases in the quantile estimates; they might balance out, or they might make things worse.
- The variance of the width of the interval will also be the sum of the variances of the quantile estimates. This could be a pretty big number, knowing that with 60 observations the "true" model variance will be large and hard to estimate.
Outside of that, it all depends on the model. Common statistical models tend to have explicit expressions for prediction intervals. In MCMC-based inference they can be inferred from the sampled values. In these cases, 60 is as good as 6,000 as long as the model works. As above, keep in mind that error propagation is not always easy to control. If these intervals are needed for serious applications, it's a good idea to generate fake data and run a sensitivity analysis.
The most generic solution is to obtaining a prediction interval is resampling. With just 60 observations, you can run full Leave-One-Out cross-validation (LOOCV). Or you can do bootstrap resampling. It might be a good idea, or the variance of the procedure will be through the roof. Again, simulation-based sensitivity analysis will help you decide.