2
$\begingroup$

I have a dataset with 130 features (1000 rows) . I want to select the best features for my classifier. I started with RFE but Its taking too long, i done this:

number_of_columns = 130

for i in range(1, number_of_columns):
    rfe = RFE(model, i)
    fit = rfe.fit(x_train, y_train)
    acc = fit.score(x_test, y_test

Because this took to long, I changed my approach, and I want to see what you think about it, is it good / correct approach.

First I did PCA, and I found out that each column participates with around 1-0.4%, except last 9 columns. Last 9 columns participate with less than 0.00001% so I removed them. Now I have 121 features.

pca = PCA() fit = pca.fit(x)

Then I split my data into train and test (with 121 features).

Then I used SelectFromModel, and I tested it with 4 different classifiers. Each classifier in SelectFromModel reduced the number of columns. I chosed the number of column that was determined by classifier that gave me the best accuracy:

model = SelectFromModel(clf, prefit=True)
#train_score = clf.score(x_train, y_train)
test_score = clf.score(x_test, y_test)
column_res = model.transform(x_train).shape

End finally I used 'RFE'. I have used number of columns that i get with 'SelectFromModel'.

rfe = RFE(model, number_of_columns)
fit = rfe.fit(x_train, y_train)
acc = fit.score(x_test, y_test)

Is this a good approach, or I did something wrong?

Also, If I got the biggest accuracy in SelectFromModel with one classifier, do I need to use the same classifier in RFE?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ In your first code block, you probably shouldn't have a loop. You ask to recursively eliminate down to just 1 feature, then start over and recursively eliminate to just 2, and so on. Just do the first one, and use the ranking_ attribute. Or, use RFECV. $\endgroup$
    – Ben Reiniger
    Aug 19 '19 at 20:50
  • $\begingroup$ PCA itself produces new features that are linear combinations of the old ones, so directly doesn't give feature selection in the same sense. But perhaps you are using it indirectly, as in stats.stackexchange.com/q/27300/232706 ? $\endgroup$
    – Ben Reiniger
    Aug 19 '19 at 20:53
1
$\begingroup$

You may have a try on Lasso (l1 penalty) which does automatic feature selection by „shrinking“ parameters. This is one of the standard approaches to data with many columns and „not so many“ rows.

sklearn.linear_model.LogisticRegression(penalty=’l1‘,...

See also this post.

Edit:

The book „Introduction to Statistical Learning“ gives a really good overview. Here are the Python code examples from the book. Section 6.6.2 covers the Lasso.

Improve this answer
$\endgroup$
5
  • $\begingroup$ And can you tell me where to use lasso, I mean, what will my output be? I have read the post that you posted, but I didnt make it to do it $\endgroup$
    – taga
    Aug 19 '19 at 20:07
  • $\begingroup$ output will be a classifier (you can predict on it, just as usual), but features are selected endogenously by the classifier. $\endgroup$
    – Peter
    Aug 19 '19 at 20:10
  • $\begingroup$ Ok, but I read that I still need to specify the number of features, I want to write a code that will give the optimal number of features and what features $\endgroup$
    – taga
    Aug 19 '19 at 20:11
  • $\begingroup$ you can just dump all features and see what lasso does. This is the great advantage of lasso. You don‘t need to decide. It happens endogenously. You may have a look at Introduction to Statistical Learning, Chapter 6, really cool github.com/JWarmenhoven/ISLR-python $\endgroup$
    – Peter
    Aug 19 '19 at 20:46
  • $\begingroup$ When I do RFE, and I get the columns/features that I should use, do I need to change my x_train, y_train, x_test and y_test (to make them only with columns that rfe selected) and run my classifier again, or not? $\endgroup$
    – taga
    Aug 20 '19 at 8:18
0
$\begingroup$

For that amount of features I use Selectbest sklearn.feature_selection.SelectKBest

To do this, I take 1/4, 1/3, 1/2, 2/3, 3/4 of all the feaures and analyze how the score used to measure the error varies.

OTHER OPTION:

I use LassoCV sklearn.linear_model.LassoCV

as follows:

kfold_on_rf = StratifiedKFold(
    n_splits=10, 
    shuffle=False, 
    random_state=SEED
)

lasso_cv = LassoCV(cv=kfold_on_rf, random_state=SEED, verbose=0)
sfm = SelectFromModel(lasso_cv)
Improve this answer
$\endgroup$
2
  • $\begingroup$ What is SelectFromModel? Your answer says you use SelectKBest but that's not part of the code snippet? $\endgroup$
    – David Waterworth
    Aug 19 '19 at 22:51
  • $\begingroup$ I have tried with Lasso, and it gives me the worst score, less than 50% $\endgroup$
    – taga
    Aug 20 '19 at 5:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.