Splitting hold-out sample and training sample only once?

Question

I have a question related to evaluating out-of-sample predictions.

For my research I want to tune two parameters related to Support Vector Machines, and use these optimized parameters to predict the hold-out sample as good as possible. To evaluate my model I obviously have to split my data in a training sample (80%) and a hold-out sample (20%). When tuning the parameters I also use 10-fold CV, but this only involves the training sample.

Now I thought that my approach is not super valid, as the 20-80 split of hold-out versus training sample is only done once, and thus might be too subject to randomness. However, I feel that this approach is used quite often, for example in http://www.sciencedirect.com/science/article/pii/S095741740400096X

In my earlier research I have done leave-one-out predictions, where I used n-1 observations and predicted the remaining observation (so that every single observation is predicted once, using the model that is trained based on only the other n-1 observations). However, in my new project this is not possible, because I then also need to tune the parameters n times, which would require too much time and would mean I lose the interpretation/plots relating to these parameter values.

In short: is it a bad thing that I only split my training and hold-out sample once? Does anyone have any comments about my current approach?

Answer 1

In general, it's a good idea to split up your data into three sets:

• Training Set (60-80% of your data)
• Cross-Validation Set (10-20% of your data)
• Test Set (10-20% of your data)

When you select a model using only a train and test set, you are selecting the model which performs the best on the test set after. This seems reasonable at first, but this is usually an overly optimistic estimate of your model's generalization error. This is because you are essentially fitting your model to another parameter $d$, where $d$ is your test set.

In order to address this problem, you can train a model on your training set and then choose the model that performs best on your cross-validation set. Now, this model is no longer fit to the test set and you can safely estimate the generalization error on the test set.

It sounds like you are already following this approach in selecting the model by choosing which one performs best on a 10-fold cross-validation on your training set. In this case, the error on your test set should be a good estimate of generalization error.

You also mentioned that randomness might affect your training due to outliers in your training data:

...as the 20-80 split of hold-out versus training sample is only done once, and thus might be too subject to randomness.

In machine learning, it is often assumed that our data is iid (independent and identically distributed) from some unknown distribution. On average, we should get data points that are representative of this distribution when we split up our data. A common practice to better ensure this is to randomly shuffle the entire dataset before splitting up your data.