Feature selection, is it possible to combine wrapper and embedded methods?

Question

I'm using neural network to predict PM10 concentration (a regression problem). Since the wrapper method is dependent on the model, so passing the neural network model that's optimized for all the features in the dataset won't work correctly (since every time the wrapper reduces the features in the dataset, the hidden layer size of the neural network must be re-tuned).

So my question is, can I use the embedded method (Lasso) as an estimator in the wrapper method?

This is the feature selection part that I' asking about:

from mlxtend.feature_selection import SequentialFeatureSelector

feature_selector = SequentialFeatureSelector(LassoCV(),
           k_features=3,
           forward=True,
           verbose=0,
           scoring='neg_mean_absolute_error',
           cv=0)
features = feature_selector.fit(X, y)
filtered_features= cols[list(features.k_feature_idx_)]
print(filtered_features)

I'm new to data science and machine learning, so I want to know if this is right or wrong.

Answer 1

Can I use Lasso as the estimator in a "wrapper" feature selection method?

It will certainly work, but the embedded feature selection of Lasso is unlikely to actually come into play in your example. The forward sequential feature selection builds the model for each candidate feature to be added to the selected feature set, starting from no features. When you only build up to three features then, the only way Lasso's feature selection will have an effect is when it determines that fewer than three variables should be kept, or if it somehow decides that one of the already-selected features should have zero coefficient. The latter would be very surprising, and the former only makes sense if your data is very simple (only two variables contribute anything significant to the predictions).

So, here Lasso is probably only really contributing as regularization, shrinking coefficients but probably not zeroing any of them out.

How should I do feature selection for a neural network?

Since Lasso is a linear model assuming feature independence, you may be handicapping the net's ability to find and use nonlinear relationships or feature interactions.

I doubt there's a "right" answer here (no free lunch sort of thing). Here are some ideas though.

1. Don't do feature selection. Count on the neural network, perhaps with heavy regularization methods (dropouts, L1 penalty especially on the first layer, etc.), to sort out what's important.
2. Do hyperparameter tuning inside the wrapper. This is very computationally expensive, but may be the most performant approach, and might be doable for smallish nets.
3. Fix the architecture inside the wrapper. You say "every time the wrapper reduces the features in the dataset, the hidden layer size of the neural network must be re-tuned," but this might not be a problem. Often enough, if a net has more neurons than it needs, several may learn nearly the same feature. (You might try to turn this into a compromised version of (2), by applying some network pruning instead of full hyperparameter tuning.)
4. Use another model inside the wrapper: one that can capture nonlinearities and feature interactions unlike a linear model, say a tree-based model. This still may fail to attribute importance to features that a network would pick up on, but it should be closer than a linear model.