1
$\begingroup$

I am trying to optimize the hyperparameters of SVM and CART with tune() function of e1071 R package, but I have a doubt. Should I tune the parameters on the training data, fit the model on the training data and then test it on the test data, or may I avoid the second step?

$\endgroup$
0
$\begingroup$

There's two common approaches here, depending on how much data you have.

If you have plenty of data, you'll probably be fine to:

  1. set part of the training set aside, and call it your validation set;

  2. start with a set of hyperparameters, train a model on your training set, evaluate performance on the validation set;

  3. repeat step 2 with different hyperparameters;

  4. pick the hyperparameters which give you the best score on the validation set;

  5. train your model on the training set and the validation set;

  6. Test your model ONCE on your test set.

Otherwise, you can do the following:

  1. start with a set of hyperparameters, evaluate your model's performance on unseen data via cross-validation on the training set;

  2. repeat step 2 with different hyperparameters;

  3. pick the hyperparameters which give you the best score on the validation set;

  4. train your model on the entire training set;

  5. Test your model ONCE on your test set.

$\endgroup$
  • $\begingroup$ I have 100 subjects, so I'll probably go with the second option. And in case I'd use CV also for testing the model should I nest the two Cross Validation? The inner one for choosing the hyperparameters and the outer one to validate the model? $\endgroup$ – schrodingercat Oct 5 '18 at 20:58
  • $\begingroup$ Yes - with such a small dataset, that is probably the best you can do $\endgroup$ – marco_gorelli Oct 8 '18 at 8:27
  • $\begingroup$ Thank you...last question: at the end I'll have a series of, for example, RMSE, one for each loop of CV. Should I choose the 'best' one or present all of them? $\endgroup$ – schrodingercat Oct 8 '18 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.