I have a question related to evaluating out-of-sample predictions.
For my research I want to tune two parameters related to Support Vector Machines, and use these optimized parameters to predict the hold-out sample as good as possible. To evaluate my model I obviously have to split my data in a training sample (80%) and a hold-out sample (20%). When tuning the parameters I also use 10-fold CV, but this only involves the training sample.
Now I thought that my approach is not super valid, as the 20-80 split of hold-out versus training sample is only done once, and thus might be too subject to randomness. However, I feel that this approach is used quite often, for example in http://www.sciencedirect.com/science/article/pii/S095741740400096X
In my earlier research I have done leave-one-out predictions, where I used n-1 observations and predicted the remaining observation (so that every single observation is predicted once, using the model that is trained based on only the other n-1 observations). However, in my new project this is not possible, because I then also need to tune the parameters n times, which would require too much time and would mean I lose the interpretation/plots relating to these parameter values.
In short: is it a bad thing that I only split my training and hold-out sample once? Does anyone have any comments about my current approach?