9
$\begingroup$

My question is three-fold

In the context of "Kernelized" support vector machines

  1. Is variable/feature selection desirable - especially since we regularize the parameter C to prevent overfitting and the main motive behind introducing kernels to a SVM is to increase the dimensionality of the problem, in such a case reducing the dimensions by parameter reduction seems counter-intuitive
  2. If the answer to the 1st question is "NO", then, On what conditions would the answer change that one should keep in mind ?
  3. Are there any good methods that have been tried to bring about feature reduction for SVMs in scikit-learn library of python - I have tried the SelectFpr method and am looking for people with experiences with different methods.
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Feature selection is always going to help unless your initial features happen to super high quality to begin with. Sklearn offers a lot of different feature selection libraries (scikit-learn.org/stable/modules/feature_selection.html) I'm partial to RFE myself. $\endgroup$
    – David
    Jul 26, 2015 at 19:58
  • 1
    $\begingroup$ You didn't mention your context - note that if it's in a business context, every feature that remains in the model is going to require maintenance at some level - there is a literal money cost involved if you have more variables, in the sense that it requires, for example, more data collection effort, DBA time and programming time. This consideration obviously doesn't apply if it's for a Kaggle comp or similar, and if using an extra 200 of the available features in the dataset gives you a 0.01% boost in performance you're in the money. $\endgroup$
    – Robert de Graaf
    Jun 2, 2016 at 11:53

1 Answer 1

Reset to default
3
$\begingroup$

Personally, I like to divide feature selection in two:

  • unsupervised feature selection
  • supervised feature selection

Unsupervised feature selection are things like clustering or PCA where you select the least redundant range of features (or create features with little redundancy). Supervised feature selection are things like Lasso where you select the features with most predictive power.

I personally usually prefer what I call supervised feature selection. So, when using a linear regression, I would select features based on Lasso. Similar methods exist to induce sparseness in neural networks.

But indeed, I don't see how I would go about doing that in a method using kernels, so you are probably better off using what I call unsupervised feature selection.

EDIT: you also asked about regularization. I see regularization as helping mostly because we work with finite samples and so the training and testing distribution will always differ somewhat, and you want your model to not overfit. I am not sure it removes the need to avoid selecting features (if you indeed have too many). I think that selecting features (or creating a smaller subset of them) helps by making the features you do have more robust and avoid the model to learn from spurious correlations. So, regularization does help, but not sure that it is a complete alternative. But I haven't thought thoroughly enough about this.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.