Feature selection - SelectKBest sklearn

Question

I would like to ask how to set paramater k in function SelectKBest for feature selection. I have now around 2300 features, so I think that default value 10 is not enough. Is there any approach, how many features choose or it is just on testing to find some compromise between accuracy and number of features?

Answer 1

This is a hyperparameter. And so this is usually approached with some sort of hyperparameters search with cross-validation, like GridSearchCV or RandomizedSearchCV. https://scikit-learn.org/stable/modules/grid_search.html

Answer 2

Since it's a hyperparameter, you should do hyperparameter tuning to determine best number of features K using GridSearch. You can also use SelectPercentile to determine percentile of features to select insted.

Example:

from sklearn.model_selection import GridSearchCV
from sklearn.feature_selection import SelectPercentile, SelectKBest
from sklearn.feature_selection import chi2
from sklearn.pipeline import Pipeline

#define your pipeline here
estimator = Pipeline(
    [ ... ,
      ("univ_select", SelectPercentile(chi2)),
      ...
      ("estimator", your_estimator())])
grid_params = {
      ...
      #define range of percentiles (or k) here
      "univ_select_percentile": [5, 10, 25, 50, 75, 100],
      ... }

# Performing Grid Search
gs = GridSearchCV(estimator,
               grid_params,
               ...)

If you want to use SelectKBest you can generate list of possible number of features using numpy:

import numpy as np
max_features = 2300
min_features = 5
step = 10
grid_params = {
      ...
      #define range of k here
      "univ_select_k": np.arange(min_features,max_features,step),
      ... }