# Does the SVM require lots of features most of the time?

So I know about the curse of dimensionality (too many features too less data).

Say I have a 3000 sample dataset, would 3 features be too less?

• It depends on the informativeness of your features. There's only one way to find out; it should not take much time with your small data set.
– Emre
Apr 20, 2017 at 18:08
• I like the implicit suggestion in your comment, "GTFO off the internet and go train the SVM" Apr 20, 2017 at 20:48

I stumbled upon some rules of thumb for dataset sizes, but not specific to the ratio of features / samples.

Also, it seems easier to guess to the "positive" side, i.e. "might it work?" then "might it fail?". I witnessed both small datasets (<1000) that functioned well with even 20-50 features, and huge datasets (~10M) that functioned well with very few (<5) features.

The stage in the processing pipeline to tackle this, is the feature selection one. I'd start with gathering all of your make-sense features and playing with different k-best values for the feature selection object to find. (ranges like: 10, 50, 100, ..)

You didn't mention a specific research environment, but I'd like to suggest scikit-learn sklearn.feature_selection.SelectKBest running inside a GridSearchCV pipeline. This way the grid object will help you choose the closest k-best featues out of a given list. Something like:

from sklearn.feature_selection import SelectKBest
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.svm import LinearSVC

pipe = []
pipe.append(SelectKBest())
pipe.append(LinearSVC())

est_pipe = make_pipeline(*pipe)
print 'all possible grid params: {0}'.format(est_pipe.get_params().keys())
param_grid = {
# some svc grid params..,
'selectkbest__k': [5, 10, 50, 100]
}

grid = GridSearchCV(est_pipe, param_grid=param_grid)
grid.fit(X, y)

• Looks like someone managed to pull the selected feature list out of an OpenCV object, using inheritance: stackoverflow.com/questions/25962349/…. The estimator there was adaboost, though
– mork
Apr 20, 2017 at 20:59
• I've come across it before, adaboost is unlikely overhead for the data I have, plus I just wanted to know if very small feature vectors can be used for large samples (this is partially due to my own very strange results) Apr 20, 2017 at 21:20
• @astroman, since the question was asked in a general way - would you consider marking this as the right answer? if not, I'd consider editing the question and tags so that others understand the context better
– mork
Apr 23, 2017 at 13:06
• No, I didn't say anything before but I'll do so now. My question was "Does the SVM require lots of features most of the time?", or "In general how many features does your SVM require?". I answered this in my own answer. My question was not, "I'm trying to do feature selection in python, I've done this much, please help me with how to move this forward from this point, using scikit-learn." or "How to do feature selection in SVM using OpenCV / C++/ python?" Apr 23, 2017 at 15:26

So I'll post an answer to my own question. For anyone who comes across this post during the feature selection / More features or less process, I dont know what you can do (well except if you're on python then mork's answer has a good way to do feature selection there) but I can tell you what NOT to do.

Do not under any circumstances ever "try" determining best features by training+testing the SVM / statistical model. That is oh this feature works because of more classification accuracy than the other one. NO. Not unless that is the only way left, dont do it. That is a way, you are free to do it, but if you can try something else please do. Dont listen to anyone who tells you to do that

How many features your problem requires depends on how many optimal features you can find. I'll leave it at that. How to find them? That is the million dollar question.

Edit:

People are getting confused. When you dont know about the accuracy of your features, it is bad practice to "train" data to see how many features your SVM needs. For that it is better you select features on the basis of some criteria set by your problem. If you want after that, then you may try feature selection techniques. But remember, reducing too many dimensions may also decrease accuracy sometimes.

• This answer lacks substance and uses a lot of words to say "I don't know." Apr 24, 2017 at 17:14
• It also says what NOT to do, like trying out classification results manually to select features all the time when there could be alternatives Apr 24, 2017 at 17:46
• Yes, but stepwise variable selection is a perfectly valid approach so that point was wrong too. Apr 24, 2017 at 18:01
• Yes it may be but not when you dont know if your features are 100% correct, it will give an erraneous result. For that you do other things, which is what this question adresses. You should only do that when you are absolutely sure about your data! Apr 24, 2017 at 20:14
• The nature of applied statistics and machine learning is that you can never know if your features are 100% or even "mostly" correct. Nearly all real-world data has noise to some degree that cannot be avoided. On the other hand, if a feature is completely useless (e.g., taken from a random number generator) then it goes without saying that you'll have to rely on something else. If that's what you were trying to say, then that's not a useful/insightful answer either. Apr 24, 2017 at 20:25