I have a relatively small dataset of 30 samples with binary labels (16 positive and 14 negative). I also have five continuous features for each these samples. I'm trying to use the support-vector classifier (SVC) for this task. I tested the performance of different feature combinations and regularization strengths in the classification task using leave-one-out cross-validation.
One odd thing that I found is that if I took feature A and used it alone for classification, I might get, say 87% classification accuracy. If I use feature B in isolation, I might get 60% classification accuracy (i.e., same as majority classifier baseline). But then combining all the features, I would get only 63% classification accuracy. This is despite performing a search across a large range of regularization strengths.
In case it matters, I'm using the sklearn SVC implementation, and varying the regularization parameter C.
Is this sort of behavior typical with an SVC classifier? I'm not too familiar with this support vector algorithms in general.