3
$\begingroup$

I have a relatively small dataset of 30 samples with binary labels (16 positive and 14 negative). I also have five continuous features for each these samples. I'm trying to use the support-vector classifier (SVC) for this task. I tested the performance of different feature combinations and regularization strengths in the classification task using leave-one-out cross-validation.

One odd thing that I found is that if I took feature A and used it alone for classification, I might get, say 87% classification accuracy. If I use feature B in isolation, I might get 60% classification accuracy (i.e., same as majority classifier baseline). But then combining all the features, I would get only 63% classification accuracy. This is despite performing a search across a large range of regularization strengths.

In case it matters, I'm using the sklearn SVC implementation, and varying the regularization parameter C.

Is this sort of behavior typical with an SVC classifier? I'm not too familiar with this support vector algorithms in general.

$\endgroup$
1
$\begingroup$

30 samples probably just isn't enough. The more features you want to use, the more samples you'll need - otherwise you'll get huge model instability and overfitting (especially with bad/useless features). With only 30 samples you'll probably get the "best" results with 1 or 2 carefully selected features. Get 100 or 200 samples, then try again with 5 features.

Also make sure you're standardizing your features - for example by removing the mean and scaling to unit variance. SVMs don't like features that are a lot larger than other features.

| improve this answer | |
$\endgroup$
  • $\begingroup$ This is a great answer, thank you! Incidentally I had just been wondering if scaling was an issue... Unfortunatel in this case it will be impractical to attain that scale of samples, but that's life... $\endgroup$ – user29968 Mar 15 '17 at 0:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.