3
$\begingroup$

I'm training a support vector machine classifier (SVM) on 100.000 observations. I'd like to try different parameter combinations (including kernel types) using cross-validation. However, it is computationally expensive to optimize over these different parameter values. Therefore, I'd like to test them on a subset of 5.000 observations on a validation set and then based on those results choose the parameters for the final model that is trained on 100.000 observations.

My question is then whether there is something intrinsic in these parameters that will make this approach problematic. More specifically, is there a parameter, where training on the full data set will require a different value for optimal performance?

I'm using the following kernels with their parameters in parentheses:

  • rbf (gamma, C)
  • poly (degree, C)
  • sigmoid (gamma, C)
  • linear (gamma).
Improve this question
$\endgroup$
3
  • $\begingroup$ In general it depends on where you are on the learning curve; if your model has converged for a sample the size of your sample then you should be fine. $\endgroup$
    – Emre
    Commented Oct 23, 2015 at 15:46
  • $\begingroup$ It hasn't converged. The best performing model on 5000 has acc of 0.74, whereas it has 0,76 on the full data. $\endgroup$
    – pir
    Commented Oct 23, 2015 at 16:40
  • $\begingroup$ I meant to say "size of your subsample". There isn't a big difference between 0.74 and 0.76. $\endgroup$
    – Emre
    Commented Oct 23, 2015 at 21:15

1 Answer 1

Reset to default
2
$\begingroup$

In theory, if you have a large enough random sample of your data set, it should be representative of the characteristics in the larger population of data that will affect the relationship between parameter values and performance. It seems to me that a $5\%$ sample of your data might be too small for what you're trying to do. When I'm performing cross-validation on a test/dev data set, I usually set aside somewhere in the neighborhood of $10\%$ for the hold-out test/evaluation data, and use $90\%$ for my development work, but, as you pointed out, this may be too big given the computational constraints you're up against. Depending on the type of application you're working on, you could look at feature distribution in your sampled data and compare it to that in the larger population. If the two sets are comparable, then you might be ok to do your dev. work on the smaller data set. I say it depends on the type of application you're developing, because many statisticians would consider this cheating, in that it's generally not good model evaluation practice to look at hold-out data at all. Here's the procedure I'd recommend:

  1. Separate out $10\%$ of your data for hold-out evaluation.
  2. Divide the remaining $90\%$ into $5\%-10\%$ parameter optimization, and the remainder for final parameter eval.
  3. Compare the feature distributions of the parameter optimization and final parameter eval data sets, if they're not comparable, redraw the samples (within the context of the $90\%$ development sample)
  4. When you have a good subsample, run your parameter optimization experiments on the small parameter optimization data set.
  5. With your final parameter settings, evaluate with cross-validation on the the $90\%$ combined parameter optimization and final parameter eval data sets.
  6. Perform final model analysis by training on the $90\%$ data set and classifying on the $10\%$ hold-out evaluation set.

It might also be worth looking into optimizing your analytics pipeline. For example, is file I/O, feature generation and feature extraction part of the the workflow you're using, or have you done all of that off-line and are just concerned with the SVM evaluation part?

Improve this answer
$\endgroup$
6
  • $\begingroup$ I'm using a validation set for comparing the different parameter combinations so I'm not cheating as I'm not looking at the test set :) But interesting thought on comparing feature distributions and taking a new sample! How would you do that in practice? What test would you use to decide whether a new sample should be used instead? $\endgroup$
    – pir
    Commented Oct 23, 2015 at 7:41
  • $\begingroup$ @felbo: If you have continuous-value features you could run KS tests, but I imagine looking at summary statistics (mean, max, etc.) and )comparing them visually using something like plot(density(dataset1$feature1)), points(density(dataset2$feature2)) should be sufficient!` $\endgroup$
    – Kyle.
    Commented Oct 23, 2015 at 11:01
  • $\begingroup$ Wouldn't it be in very rare cases that e.g. the mean/std will be different I have a sample of 5000? $\endgroup$
    – pir
    Commented Oct 23, 2015 at 12:23
  • 2
    $\begingroup$ I feel like @Kyle answer is true for kernel type, degree and possibly even gamma, but isn't true for C. C is the regularization parameter which is used to tweak models when high variance becomes an issue. Another well-known tweak to adjust for high variance is adding more data. So you should expect that you will be able to turn down C a bit as you add the extra data. I would suggest following Kyle's recommendation then try a couple more tweaks with backing off on C with closer to the full set. $\endgroup$
    – AN6U5
    Commented Jan 21, 2016 at 4:26
  • 1
    $\begingroup$ @AN6U5 good correction! I agree. $\endgroup$
    – Kyle.
    Commented Jan 21, 2016 at 4:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.