We intend to model data with non-parametric covariate splines and we would like to understand the uncertainty of the parameter estimates/response estimates.

Currently, we use cross-validation to model the optimal smoothness of our spline models using penalized likelihoods. Moreover, we use bootstrap around the entire procedure to understand the sensitivity in parameter estimates. NB: for each bootstrap iteration, we use cross-validation to find the optimal smoothness for each bootstrap resample.

I currently have the following problem with this methodology:

  • If you bootstrap everything and then perform a random k-fold cross validation, there will be observations that occur in both test and training datasets. This leads to overfitting. NB: this effect would be worse for larger k.

What is current state of the art for a problem like this? We are considering the following improvements:

  • Only do cross-validation for fitting optimal smoothness on original data. Use this smoothness parameter for the remaining bootstraps. NB: this methodology would not be able to assess the uncertainty of the smoothness parameter.
  • For each bootstrap, first divide the data into folds and then bootstrap within these folds such that for the k-fold cross validation there is no overlap in test and training sets. Note: for n-fold cross validation (n = length of data), this would imply that bootstrap does nothing as there is only one way to do this. Hence, this would underestimate the real variability.

Finally, I am also aware of the concept of bootstrap-cross validation or 632 cross-validation, where you assign an observation with probability 1-exp(-1) = 63.2% to the training set and otherwise assign it to the test set. And then fit your model and repeat this partitioning multiple times. This type of cross-validation could be used at any stage above instead of k-fold cross validation.

What is the current state of the art? I cannot find references that discuss this type of problem.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.