I have a question regarding steps on which a specific resample method should be used in general.
As far as I know:
- out-of-bag bootstrap is the resample method with replacement, which has lower variance and higher bias compared to the cross-validation method.
- cross-validation is the resample method without replacement, which has higher variance and lower bias compared to the out-of-bag bootstrap method.
Following descriptions above, isn't it better (in general) to tune models with out-of-bag bootstrap resampling and use cross-validation only for final scoring of the tuned models?
Tuning is the process, which should answer the question: What is the best model configuration according to the specific evaluation metric? So, we're not interested in finding exact performance measure, but we would rather focus only on the comparison of different configurations. Lower variance means a lower risk of choosing a model with a high mean performance, although few sets have a very poor performance.
On the other hand, cross-validation performed on the tuned model gives us an answer to the question: What is the approximate performance of the tuned model?