I hope you don't mind me asking a few questions in one - they're all related somewhat. I'm working on a simple classification problem (Titanic, you guessed it) and I'm trying to grind out the last little bit of accuracy I can by restricting the features used in the model to only the most effective ones (read: feature selection).

I already know how to filter out features by their chi-squared score or their mutual information with the target variable. One thing I noticed is that I have a couple of sets of engineered features which are all somewhat correlated with each other. I'd prefer not to use all of them in certain models, but some seem to be better for some estimators, and others better for other estimators. For example, if $F := {1, ..., n}$ are the labels of my features, then I have a subset $S\subset F$ of features such that the pairwise correlation of each feature in $S$ is $>2/3$. Ideally, I want my model to use only one of the features from $S$.

So, I want to iterate over each feature subset which includes at most one feature in a given subset. In other words, given the definitions above, I'd like to iterate over all feature subsets in the family $\mathcal{F}$: $$ \mathcal{F}:=\{F\backslash S\}\cup\{F\backslash S\ \cup \{i\}\ |\ i\in S\} $$ For each $G\in\mathcal{F}$, I train the model on $G$ and obtain a cross-validation accuracy. The subset $G^*$ with the highest accuracy model wins. I hope that makes sense, I feel like I haven't explained it terribly well.

Question 1: is there a name for this procedure? I can't conceptually compare it with the other feature selection methods I'm aware of.

Question 2: is this even a good idea? I know some models suffer more from the curse of dimensionality than others (e.g. SVCs), so I'd like to keep the feature space low-dimensional. However, I'm also fairly certain that some classifiers (e.g., again, SVCs) aren't weakened by the presence of correlated features, so I wonder why even bother?

Also, these highly correlated features $S$ are essentially just minor modifications of each other (not as simple as being linear functions of each other, more like one is a binned version of another continuous feature, but the binned feature works better for some models than the continuous feature). Is this bad practice? Should I just filter out the least relevant features from $S$ beforehand and call it a day (e.g. choose only that feature from $S$ with the highest mutual information gain with the target variable)?

Question 3: assuming this is a fine idea, what's the best way to implement it? I'm using scikit-learn, and I currently have setup a GridSearchCV(estimator, param_grid). There are two ways I see to approach this:

  1. Wrap estimator with a custom estimator that iterates over the desired feature subsets. Something like GridSearchCV(FeatureSelector(estimator, one_of=["feature1", "feature2", ...]), param_grid).

  2. Use a pipeline with a transformer that restricts the input to the given features, then list the desired feature subsets in the param_grid. In pseudocode, sometihng like

    Pipeline(steps = [('rct', restrict_columns_transformer), ('model', estimator)]), 
    param_grid = {'rct__features' : [#every feature subset...],

Is there a better way I'm not thinking of?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.