We intend to model data with non-parametric covariate splines and we would like to understand the uncertainty of the parameter estimates/response estimates.
Currently, we use cross-validation to model the optimal smoothness of our spline models using penalized likelihoods. Moreover, we use bootstrap around the entire procedure to understand the sensitivity in parameter estimates. NB: for each bootstrap iteration, we use cross-validation to find the optimal smoothness for each bootstrap resample.
I currently have the following problem with this methodology:
- If you bootstrap everything and then perform a random k-fold cross validation, there will be observations that occur in both test and training datasets. This leads to overfitting. NB: this effect would be worse for larger k.
What is current state of the art for a problem like this? We are considering the following improvements:
- Only do cross-validation for fitting optimal smoothness on original data. Use this smoothness parameter for the remaining bootstraps. NB: this methodology would not be able to assess the uncertainty of the smoothness parameter.
- For each bootstrap, first divide the data into folds and then bootstrap within these folds such that for the k-fold cross validation there is no overlap in test and training sets. Note: for n-fold cross validation (n = length of data), this would imply that bootstrap does nothing as there is only one way to do this. Hence, this would underestimate the real variability.
Finally, I am also aware of the concept of bootstrap-cross validation or 632 cross-validation, where you assign an observation with probability 1-exp(-1) = 63.2% to the training set and otherwise assign it to the test set. And then fit your model and repeat this partitioning multiple times. This type of cross-validation could be used at any stage above instead of k-fold cross validation.
What is the current state of the art? I cannot find references that discuss this type of problem.
