Does the SVM require lots of features most of the time?

Question

So I know about the curse of dimensionality (too many features too less data).

Say I have a 3000 sample dataset, would 3 features be too less?

Answer 1

I stumbled upon some rules of thumb for dataset sizes, but not specific to the ratio of features / samples.

Also, it seems easier to guess to the "positive" side, i.e. "might it work?" then "might it fail?". I witnessed both small datasets (<1000) that functioned well with even 20-50 features, and huge datasets (~10M) that functioned well with very few (<5) features.

The stage in the processing pipeline to tackle this, is the feature selection one. I'd start with gathering all of your make-sense features and playing with different k-best values for the feature selection object to find. (ranges like: 10, 50, 100, ..)

You didn't mention a specific research environment, but I'd like to suggest scikit-learn sklearn.feature_selection.SelectKBest running inside a GridSearchCV pipeline. This way the grid object will help you choose the closest k-best featues out of a given list. Something like:

from sklearn.feature_selection import SelectKBest
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import GridSearchCV
from sklearn.svm import LinearSVC

pipe = []
pipe.append(SelectKBest())
pipe.append(LinearSVC())

est_pipe = make_pipeline(*pipe)
print 'all possible grid params: {0}'.format(est_pipe.get_params().keys())
param_grid = {
  # some svc grid params..,
  'selectkbest__k': [5, 10, 50, 100]
}

grid = GridSearchCV(est_pipe, param_grid=param_grid)
grid.fit(X, y)

Answer 2

So I'll post an answer to my own question. For anyone who comes across this post during the feature selection / More features or less process, I dont know what you can do (well except if you're on python then mork's answer has a good way to do feature selection there) but I can tell you what NOT to do.

Do not under any circumstances ever "try" determining best features by training+testing the SVM / statistical model. That is oh this feature works because of more classification accuracy than the other one. NO. Not unless that is the only way left, dont do it. That is a way, you are free to do it, but if you can try something else please do. Dont listen to anyone who tells you to do that

How many features your problem requires depends on how many optimal features you can find. I'll leave it at that. How to find them? That is the million dollar question.

Edit:

People are getting confused. When you dont know about the accuracy of your features, it is bad practice to "train" data to see how many features your SVM needs. For that it is better you select features on the basis of some criteria set by your problem. If you want after that, then you may try feature selection techniques. But remember, reducing too many dimensions may also decrease accuracy sometimes.