When/how should I use the validation set for hyper-parameter sweeps for neural networks?

Question

I know similar questions have been asked so many times, but I couldn't find the answer to this one particularly, at least not in a way that satisfied me.

I am very confused about how to use validation sets. I know they are used to perform hyper parameter sweeps but I'm not quite sure in what way. For example, suppose I am trying to decide between a neural network that has 1 hidden layer and one which has 2, and I have a 50/25/25 train/validation/test split of my data. I will not be performing k-fold CV for reasons which are inherent to my problem.

Should I be using the validation set during training (to prevent overfitting) and the same validation set after the fact (once when the models are completely trained) to determine which model is better? When do I use the validation set to compare models?

And then once I get to the test set, I know I am supposed to train on the training data and validation set together. But should I actually keep the validation set separate from the training set during training of the finalized model to prevent overfitting? Why would I use the separate validation set during training of the individual models to prevent overfitting, but not use it for that purpose (and make it more training data) on the final model?

Answer 1

Let's take a step back and look at why we make these splits:

Model selection: estimating the performance of different models in order to choose the best one.

Model assessment: having chosen a final model, estimating its predic- tion error (generalization error) on new data.

(Source: "The Elements of Statistical Learning - Data Mining, Inference, and Prediction", Hastie et al)

For model selection you use the validation set and for model assessment you use the test set.

---

Accordingly, a straight forward approach would be this:

1. Split data into train/valid/test sets
2. Train models on train dataset
3. Compare models on valid dataset
4. Repeat steps 2 and 3 until you meet your personal stopping criterion (e.g. performance sufficiently)
5. Select your final model and retrain it on the train and valid dataset
6. Evaluate the performance of your selected and retrained model on the test dataset

---

Here is an example with an SVM taken from "Introduction to Machine Learning with Python" by Mueller and Guido:

from sklearn.svm import SVC

# split data into train+validation set and test set
X_trainval, X_test, y_trainval, y_test = train_test_split(
iris.data, iris.target, random_state=0)

# split train+validation set into training and validation sets
X_train, X_valid, y_train, y_valid = train_test_split(
X_trainval, y_trainval, random_state=1)
print("Size of training set: {} size of validation set: {} size of test set:"
" {}\n".format(X_train.shape[0], X_valid.shape[0], X_test.shape[0]))
best_score = 0
for gamma in [0.001, 0.01, 0.1, 1, 10, 100]:
for C in [0.001, 0.01, 0.1, 1, 10, 100]:

# for each combination of parameters, train an SVC
svm = SVC(gamma=gamma, C=C)
svm.fit(X_train, y_train)

# evaluate the SVC on the test set
score = svm.score(X_valid, y_valid)

# if we got a better score, store the score and parameters
if score > best_score:
best_score = score
best_parameters = {'C': C, 'gamma': gamma}

# rebuild a model on the combined training and validation set,
# and evaluate it on the test set
svm = SVC(**best_parameters)
svm.fit(X_trainval, y_trainval)
test_score = svm.score(X_test, y_test)
print("Best score on validation set: {:.2f}".format(best_score))
print("Best parameters: ", best_parameters)
print("Test set score with best parameters: {:.2f}".format(test_score))