3
$\begingroup$

I have a wine dataset with 13 features that indicates 3 different wine classes (target), and k-NN, SVM with linear kernel and SVM with rbf kernel algorithms to be tried with this dataset.

My goal is to obtain the best classification accuracy, and to obtain this accuracy:

  1. Which classification algorithm (kNN, SVM with linear kernel or SVM with rbf kernel) should I choose?

  2. Among all the features, which ones of them (based on backward elimination, maybe according to p-values) should be chosen?

I have thought of using GridSearchCV with 3 estimators for the algorithms above. But in this case, the problem is feature selection part as you guess. Is there any optimal way to achieve both? Thanks!

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Maybe you should incorporate PCA into your analysis. PCA with a steep curve will a small feature space can tell a lot about your data where a linear sloped plotting of PCA means all features are very relevant. $\endgroup$
    – Michael Hearn
    Nov 12, 2019 at 21:28
  • $\begingroup$ @MichaelHearn PCA is a dimensionality reduction algorithm. Although it may help with high dimensional data, it doesn't tell anything about feature importance, and it's important to note that the two are not strictly the same: one may want to know which features tell something, which are 'neutral', and which are just noise which make the algorithm perform worse than if they weren't there. That can't be achieved with PCA. $\endgroup$
    – 89f3a1c
    Nov 12, 2019 at 22:24
  • 1
    $\begingroup$ I was referring to plotting 'variance explained by principal components'. If they plot variance explained, say after removing a feature or group of features they can clearly see the effects on the slope of the principal components ability to determine class which I believe would be beneficial for this task. I do agree there are other ways of doing this though. $\endgroup$
    – Michael Hearn
    Nov 12, 2019 at 22:45
  • $\begingroup$ You could use Ridge classifier from scikit. However, I know this probably not what you are looking for. Other than that you may have a look on scikit.ensemble.ExtraTreesClassifier which is building randomized decision trees based on your input and output data to assess the predictive power of your features. $\endgroup$
    – Maeaex1
    Nov 13, 2019 at 13:07

1 Answer 1

Reset to default
2
$\begingroup$

If you want to know what the feature importance of a dataset is, you can obtain it by training a random forest. After training the random forest, you can access the feature importance which is valid for any algorithm.

Note that the feature importance of other similar algorithms, such as some boosting algorithms, is strictly related to those algorithms; such thing is not true for random forest.

Hope this helps!

EDIT1:

The reason I said random forests give a sort of universal feature importance is that a rf is based on a lot of smaller decision trees in which each one uses bootstrap from the train set and a subset of attributes taken randomly. The bootstrap tries to avoid overfitting, whereas the subset of attributes helps determine which ones are the most important. A rf is capable of giving the importance of each feature averaging the oob precision from the trees that use the attribute. When a feature is a great predictor, the trees that use it have better results that those that don't use it. With hundreds or thousands of trees, a rf can have a very good opinion on the predicting capacity of each attribute.

Note that each tree in a rf is a decision tree which bases the split on Gini impurity/information gain (entropy), or variance in the case of regression. This naturally selects the features that matter most in each case.

Here, an article which explores alternative ways of studying feature importance. Some more things are said about rf, but it's not limited to that method, so it may be useful for anyone reading this post. https://towardsdatascience.com/explaining-feature-importance-by-example-of-a-random-forest-d9166011959e

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ Citation needed that random forests' feature importance are somehow universal (moreso than any other model that provides an implicit feature importance). $\endgroup$
    – Ben Reiniger ♦
    Nov 12, 2019 at 22:30
  • $\begingroup$ @BenReiniger Indeed, especially as there are several ways to compute feature importance from a random forest, and of course, not all give the same results (see this blog post for instance: towardsdatascience.com/…). I think this answer is misleading. $\endgroup$
    – Romain Reboulleau
    Nov 13, 2019 at 12:37
  • 2
    $\begingroup$ @BenReiniger you were right, thanks for pointing it out. Added a brief explanation on why I wrote that. Hope it's less misleading now. $\endgroup$
    – 89f3a1c
    Nov 19, 2019 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.