# Feature selection, is it possible to combine wrapper and embedded methods?

I'm using neural network to predict PM10 concentration (a regression problem). Since the wrapper method is dependent on the model, so passing the neural network model that's optimized for all the features in the dataset won't work correctly (since every time the wrapper reduces the features in the dataset, the hidden layer size of the neural network must be re-tuned).

So my question is, can I use the embedded method (Lasso) as an estimator in the wrapper method?

from mlxtend.feature_selection import SequentialFeatureSelector

feature_selector = SequentialFeatureSelector(LassoCV(),
k_features=3,
forward=True,
verbose=0,
scoring='neg_mean_absolute_error',
cv=0)
features = feature_selector.fit(X, y)
filtered_features= cols[list(features.k_feature_idx_)]
print(filtered_features)


I'm new to data science and machine learning, so I want to know if this is right or wrong.

• Ben's answer is great, but I'd like to state that many wrapper methods (like Boruta, recursive feature elimination/null importances, etc.) require feature importance scores in some fashion, which in turn can only be computed with methods that offer some sort of embedded feature selection (most tree based learners, Lasso/Elastic net, etc.) – aranglol Sep 4 '19 at 5:23
• @aranglol Thanks for the comment. I tried most of the generalized non-parametric estimators (since I'm trying to avoid feeding the wrapper a parametric model) like Lasso, Ridge, ElasticNet.. etc, and most of them gave very similar results. – M. Grimm Sep 4 '19 at 15:55

# Can I use Lasso as the estimator in a "wrapper" feature selection method?

It will certainly work, but the embedded feature selection of Lasso is unlikely to actually come into play in your example. The forward sequential feature selection builds the model for each candidate feature to be added to the selected feature set, starting from no features. When you only build up to three features then, the only way Lasso's feature selection will have an effect is when it determines that fewer than three variables should be kept, or if it somehow decides that one of the already-selected features should have zero coefficient. The latter would be very surprising, and the former only makes sense if your data is very simple (only two variables contribute anything significant to the predictions).

So, here Lasso is probably only really contributing as regularization, shrinking coefficients but probably not zeroing any of them out.

# How should I do feature selection for a neural network?

Since Lasso is a linear model assuming feature independence, you may be handicapping the net's ability to find and use nonlinear relationships or feature interactions.

I doubt there's a "right" answer here (no free lunch sort of thing). Here are some ideas though.

1. Don't do feature selection. Count on the neural network, perhaps with heavy regularization methods (dropouts, L1 penalty especially on the first layer, etc.), to sort out what's important.
2. Do hyperparameter tuning inside the wrapper. This is very computationally expensive, but may be the most performant approach, and might be doable for smallish nets.
3. Fix the architecture inside the wrapper. You say "every time the wrapper reduces the features in the dataset, the hidden layer size of the neural network must be re-tuned," but this might not be a problem. Often enough, if a net has more neurons than it needs, several may learn nearly the same feature. (You might try to turn this into a compromised version of (2), by applying some network pruning instead of full hyperparameter tuning.)
4. Use another model inside the wrapper: one that can capture nonlinearities and feature interactions unlike a linear model, say a tree-based model. This still may fail to attribute importance to features that a network would pick up on, but it should be closer than a linear model.
• Thanks for the detailed explanation! Actually using the Lasso as the estimator reduced the number of features and gave better performance than all the features in the neural network (altho as you said it's a linear model so I doubt the features it selected are optimal). Can you tell me how can I do (2) please? I tried altering the source code of the SFS to allow internal NN parameter tuning but I couldn't do it right. – M. Grimm Sep 4 '19 at 15:50
• I already tried a tree-based model like XGBoost but it didn't give good results. And I tried feeding the original NN model to the wrapper and it gave close results to the Lasso. – M. Grimm Sep 4 '19 at 15:53
• Re: Lasso, my point was the embedded feature-selection aspect of Lasso likely doesn't play a role, when using it in a wrapper feature-selection for only three features; I'd expect Ridge to produce similar results. On (2), SFS is built to use sklearn pipelines, so using e.g. RandomizedSearchCV should work, if your NN framework has or can be coerced into using a sklearn-style API. Your second comment is quite interesting; it suggests that (4) isn't great in your situation, and that (3) doesn't outperform (0) [OP approach] significantly. – Ben Reiniger Sep 5 '19 at 13:58
• Yep, Ridge, Elastic Net and other generalized estimators gave very similar results. Perhaps my data is simple like you stated "the former only makes sense if your data is very simple" so only 2 or 3 predictors contribute the most to the prediction process, that's why the wrapper with NN or with Lasso gave similar results (which improves the prediction in both cases). Thanks a lot for enlightening me. – M. Grimm Sep 5 '19 at 20:15
• Also Linear Regression performs as well as NN on my data. That could be why those linear estimators give good results right? – M. Grimm Sep 5 '19 at 20:57