# Feature selection - SelectKBest sklearn

I would like to ask how to set paramater k in function SelectKBest for feature selection. I have now around 2300 features, so I think that default value 10 is not enough. Is there any approach, how many features choose or it is just on testing to find some compromise between accuracy and number of features?

• Welcome to this site! How about plotting (model accuracy, k) to find a suitable k? for k = 1, 2, 4, 8, 16, 32, ... – Esmailian Mar 28 '19 at 19:21
• The kind of plot @Esmailian is suggesting is something called an elbow plot. I would also recommend that, using the k spacing @Esmailian suggests. – HS-nebula Apr 12 '19 at 20:40

This is a hyperparameter. And so this is usually approached with some sort of hyperparameters search with cross-validation, like GridSearchCV or RandomizedSearchCV. https://scikit-learn.org/stable/modules/grid_search.html

Since it's a hyperparameter, you should do hyperparameter tuning to determine best number of features K using GridSearch. You can also use SelectPercentile to determine percentile of features to select insted.

Example:

from sklearn.model_selection import GridSearchCV
from sklearn.feature_selection import SelectPercentile, SelectKBest
from sklearn.feature_selection import chi2
from sklearn.pipeline import Pipeline

estimator = Pipeline(
[ ... ,
("univ_select", SelectPercentile(chi2)),
...
("estimator", your_estimator())])
grid_params = {
...
#define range of percentiles (or k) here
"univ_select_percentile": [5, 10, 25, 50, 75, 100],
... }

# Performing Grid Search
gs = GridSearchCV(estimator,
grid_params,
...)


If you want to use SelectKBest you can generate list of possible number of features using numpy:

import numpy as np
max_features = 2300
min_features = 5
step = 10
grid_params = {
...
#define range of k here
"univ_select_k": np.arange(min_features,max_features,step),
... }