Tuning svm and cart hyperparameters

Question

I am trying to optimize the hyperparameters of SVM and CART with tune() function of e1071 R package, but I have a doubt. Should I tune the parameters on the training data, fit the model on the training data and then test it on the test data, or may I avoid the second step?

Answer 1

There's two common approaches here, depending on how much data you have.

If you have plenty of data, you'll probably be fine to:

1. set part of the training set aside, and call it your validation set;

2. start with a set of hyperparameters, train a model on your training set, evaluate performance on the validation set;

3. repeat step 2 with different hyperparameters;

4. pick the hyperparameters which give you the best score on the validation set;

5. train your model on the training set and the validation set;

6. Test your model ONCE on your test set.

Otherwise, you can do the following:

1. start with a set of hyperparameters, evaluate your model's performance on unseen data via cross-validation on the training set;

2. repeat step 2 with different hyperparameters;

3. pick the hyperparameters which give you the best score on the validation set;

4. train your model on the entire training set;

5. Test your model ONCE on your test set.