0
$\begingroup$

How to tune the C 'BoxConstraint' hyperparameter in soft margin SVM to get the best optimal value?

$\endgroup$
4
$\begingroup$

The easiest way to tune a single hyperparameter is to use what is called the elbow method. Do the following:

  • Define a range of C you want to try, i.e C = [1.0, 1.5, 2.0, ...]
  • Loop over all values of C in your range
    • Train a new model with the current value of C
    • Evaluate each model on the validation set and store the results
  • Plot your metric over over your range of C's

If you didn't choose a too narrow range you should visually see your optimal value of C by finding the value that either minimize or maximize your metric.

enter image description here

If you need to tune C alongside other hyperparameters you can use strategies such as grid search, randomized search or bayesian search.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I think you mean Validation Set. One should never use the test set for hyperparameter tuning! $\endgroup$ – Pedro Henrique Monforte Apr 9 '19 at 21:29
  • 1
    $\begingroup$ You are correct. Thank you for noticing! Edited and corrected. $\endgroup$ – Simon Larsson Apr 9 '19 at 21:35
  • $\begingroup$ I am using 10 fold cross-validation with test and train partitions. so I don't have validation set. is it applicable to use it? $\endgroup$ – gin Apr 10 '19 at 8:12
  • 1
    $\begingroup$ there is 'OptimizeHyperparameters' function in matlab that uses Bayesian Optimization to generate good parameters. Do you think it is suitable for the linear soft margin SVM ? mathworks.com/help/stats/… $\endgroup$ – gin Apr 10 '19 at 8:17
  • $\begingroup$ Yes, it is still applicable but any time you want to perform tuning you need to set aside some validation data to do it on. This applies regardless of the method you use to tune. $\endgroup$ – Simon Larsson Apr 10 '19 at 8:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.