2
$\begingroup$

I have a question related to evaluating out-of-sample predictions.

For my research I want to tune two parameters related to Support Vector Machines, and use these optimized parameters to predict the hold-out sample as good as possible. To evaluate my model I obviously have to split my data in a training sample (80%) and a hold-out sample (20%). When tuning the parameters I also use 10-fold CV, but this only involves the training sample.

Now I thought that my approach is not super valid, as the 20-80 split of hold-out versus training sample is only done once, and thus might be too subject to randomness. However, I feel that this approach is used quite often, for example in http://www.sciencedirect.com/science/article/pii/S095741740400096X

In my earlier research I have done leave-one-out predictions, where I used n-1 observations and predicted the remaining observation (so that every single observation is predicted once, using the model that is trained based on only the other n-1 observations). However, in my new project this is not possible, because I then also need to tune the parameters n times, which would require too much time and would mean I lose the interpretation/plots relating to these parameter values.

In short: is it a bad thing that I only split my training and hold-out sample once? Does anyone have any comments about my current approach?

$\endgroup$
  • $\begingroup$ Can you explain more about what you mean by "too subject to randomness"? $\endgroup$ – gingermander Dec 19 '17 at 16:09
  • $\begingroup$ Well, that the error that I compute it not representative for the actual performance of my model. Since I only once pick a hold-out sample and use that to evaluate to the model that I made based on my training sample (which I also only pick once of course), it could be that my hold-out contains many outliers so that the accuracy is lower than when these outliers would have been in the training set and thus do not need to be predicted. Of course then the model would slightly change as well, because the outliers now influence the model that I make (cause they are included in the training set). $\endgroup$ – dubvice Dec 19 '17 at 17:33
2
$\begingroup$

In general, it's a good idea to split up your data into three sets:

  • Training Set (60-80% of your data)
  • Cross-Validation Set (10-20% of your data)
  • Test Set (10-20% of your data)

When you select a model using only a train and test set, you are selecting the model which performs the best on the test set after. This seems reasonable at first, but this is usually an overly optimistic estimate of your model's generalization error. This is because you are essentially fitting your model to another parameter $d$, where $d$ is your test set.

In order to address this problem, you can train a model on your training set and then choose the model that performs best on your cross-validation set. Now, this model is no longer fit to the test set and you can safely estimate the generalization error on the test set.

It sounds like you are already following this approach in selecting the model by choosing which one performs best on a 10-fold cross-validation on your training set. In this case, the error on your test set should be a good estimate of generalization error.

You also mentioned that randomness might affect your training due to outliers in your training data:

...as the 20-80 split of hold-out versus training sample is only done once, and thus might be too subject to randomness.

In machine learning, it is often assumed that our data is iid (independent and identically distributed) from some unknown distribution. On average, we should get data points that are representative of this distribution when we split up our data. A common practice to better ensure this is to randomly shuffle the entire dataset before splitting up your data.

$\endgroup$
  • $\begingroup$ Yes, I indeed select the parameters using 10-fold cross validation on my training set. I suppose what you call 'test-set' is equivalent to what I called 'hold-out sample' in my question. Thanks for your opinion. $\endgroup$ – dubvice Dec 19 '17 at 17:36
  • $\begingroup$ In response to your edit: you mention that ''On average, we should get data points that are representative of this distribution when we split up our data''. However, if I only do the split once, the 'on average' thing does not really hold does it? Also, I do randomly sample from the data to get the 80-20 split. But still its only one sample, so that's why I thought it would be sensitive to randomness. $\endgroup$ – dubvice Dec 19 '17 at 19:09
  • $\begingroup$ Think of it this way. If you randomly sample a single data point from your dataset, most of the time (on average) it will not be an outlier. If you continue selecting points at random then most of the time you will find that they are representative of the distribution of your data. This means that if you are randomly selecting points from your data to place in train and test sets, it is unlikely that a significant number of outliers will only be placed in the training set. $\endgroup$ – gingermander Dec 19 '17 at 19:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.