I am working on a problem with too many features and training my models takes way too long. I implemented a forward selection algorithm to choose features.

However, I was wondering does scikit-learn have a forward selection/stepwise regression algorithm?

  • $\begingroup$ I created my own class for that but very surprised that sklearn doesn't have that. $\endgroup$
    – Maksud
    Aug 9, 2014 at 14:36
  • 1
    $\begingroup$ Using hypothesis tests is a terrible method of feature selection. You'll have to do a lot of them and, of course, you'll get a lot of false positives and negatives. $\endgroup$ Jun 3, 2016 at 8:51

8 Answers 8


No, scikit-learn does not seem to have a forward selection algorithm. However, it does provide recursive feature elimination, which is a greedy feature elimination algorithm similar to sequential backward selection. See the documentation here

  • 4
    $\begingroup$ Good suggestion, but the problem w/ the sci-kit implementation is that the feature importance is quantified by the model coefficients, i.e. if the model has coef_ interface. This would rule out tree based method etc. However, I think what @Maksud asked for is what is described in "An Introduction to statistical learning" by James in which features are recursively added/removed by their importance which his quantified by validation set accuracy. This allows feature selection across all types of models, not the just linear parametric ones. $\endgroup$
    – eggie5
    Apr 18, 2017 at 21:38
  • $\begingroup$ sci-kit actually also supports (some?) tree based method for feature elimination via feature importance (see e.g., the RandomForestClassifier). But more general methods like SVMs are indeed not supported. $\endgroup$ Apr 1, 2020 at 17:11
  • $\begingroup$ This answer is outdated. There is indeed a forward selection implementation in sklearn. scikit-learn.org/stable/modules/generated/… $\endgroup$
    – anonuser01
    Apr 26, 2021 at 0:24
  • 1
    $\begingroup$ @anonuser01 What you suggest is not correct. The forward stepwise selection does not require n_features_to_select to be set beforehand, but the sklearn's sequentialfeatureselector (the thing that you linked) does. So these two thing are different $\endgroup$
    – Eiffelbear
    Oct 9, 2021 at 8:45

Scikit-learn indeed does not support stepwise regression. That's because what is commonly known as 'stepwise regression' is an algorithm based on p-values of coefficients of linear regression, and scikit-learn deliberately avoids inferential approach to model learning (significance testing etc). Moreover, pure OLS is only one of numerous regression algorithms, and from the scikit-learn point of view it is neither very important, nor one of the best.

There are, however, some pieces of advice for those who still need a good way for feature selection with linear models:

  1. Use inherently sparse models like ElasticNet or Lasso.
  2. Normalize your features with StandardScaler, and then order your features just by model.coef_. For perfectly independent covariates it is equivalent to sorting by p-values. The class sklearn.feature_selection.RFE will do it for you, and RFECV will even evaluate the optimal number of features.
  3. Use [an implementation][1] of forward selection by adjusted $R^2$ that works with statsmodels.
  4. Do brute-force forward or backward selection to maximize your favorite metric on cross-validation (it could take approximately quadratic time in number of covariates). A scikit-learn compatible mlxtend package [supports][2] this approach for any estimator and any metric.
  5. If you still want vanilla stepwise regression, it is easier to base it on statsmodels, since this package calculates p-values for you. A basic forward-backward selection could look like this:

    from sklearn.datasets import load_boston
    import pandas as pd
    import numpy as np
    import statsmodels.api as sm

    data = load_boston()
    X = pd.DataFrame(data.data, columns=data.feature_names)
    y = data.target

    def stepwise_selection(X, y, 
                           threshold_out = 0.05, 
        """ Perform a forward-backward feature selection 
        based on p-value from statsmodels.api.OLS
            X - pandas.DataFrame with candidate features
            y - list-like with the target
            initial_list - list of features to start with (column names of X)
            threshold_in - include a feature if its p-value < threshold_in
            threshold_out - exclude a feature if its p-value > threshold_out
            verbose - whether to print the sequence of inclusions and exclusions
        Returns: list of selected features 
        Always set threshold_in < threshold_out to avoid infinite looping.
        See https://en.wikipedia.org/wiki/Stepwise_regression for the details
        included = list(initial_list)
        while True:
            # forward step
            excluded = list(set(X.columns)-set(included))
            new_pval = pd.Series(index=excluded)
            for new_column in excluded:
                model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included+[new_column]]))).fit()
                new_pval[new_column] = model.pvalues[new_column]
            best_pval = new_pval.min()
            if best_pval < threshold_in:
                best_feature = new_pval.idxmin()
                if verbose:
                    print('Add  {:30} with p-value {:.6}'.format(best_feature, best_pval))

            # backward step
            model = sm.OLS(y, sm.add_constant(pd.DataFrame(X[included]))).fit()
            # use all coefs except intercept
            pvalues = model.pvalues.iloc[1:]
            worst_pval = pvalues.max() # null if pvalues is empty
            if worst_pval > threshold_out:
                worst_feature = pvalues.idxmax()
                if verbose:
                    print('Drop {:30} with p-value {:.6}'.format(worst_feature, worst_pval))
            if not changed:
        return included

    result = stepwise_selection(X, y)

    print('resulting features:')

This example would print the following output:

    Add  LSTAT                          with p-value 5.0811e-88
    Add  RM                             with p-value 3.47226e-27
    Add  PTRATIO                        with p-value 1.64466e-14
    Add  DIS                            with p-value 1.66847e-05
    Add  NOX                            with p-value 5.48815e-08
    Add  CHAS                           with p-value 0.000265473
    Add  B                              with p-value 0.000771946
    Add  ZN                             with p-value 0.00465162
    resulting features:
    ['LSTAT', 'RM', 'PTRATIO', 'DIS', 'NOX', 'CHAS', 'B', 'ZN']

  [1]: http://planspace.org/20150423-forward_selection_with_statsmodels/
  [2]: http://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/
  • 2
    $\begingroup$ The posted forward stepwise regression code does not function correctly. It should give identical results to backwards stepwise regression, but it does not. It is returning factors with p-values that are higher than the threshold when you rerun the regression. I also ran the same dataset with STATA and the same thresholds using backwards stepwise and obtain materially different results. Basically, don't use it. I'm going to write my own backwards stepwise regression code using his template. $\endgroup$ Apr 27, 2019 at 12:38
  • 5
    $\begingroup$ Forward and backward stepwise regressions are by no means guaranteed to converge to the same solution. And if you noticed a bug in my solution, please attach the code to reproduce it. $\endgroup$
    – David Dale
    Apr 27, 2019 at 14:18
  • $\begingroup$ Is it possible to change it into stepwise selection for Logistic Regresion? I have tried changing sm.OLS into sm.Logit but it produced an error $\endgroup$
    – AAAA
    Sep 2, 2020 at 8:01
  • $\begingroup$ This example is no longer working with python 3.7, the Series need dtype arg (e.g. pd.Series(index=excluded, dtype='float64') , and OLS function need to be updated. $\endgroup$
    – Ibrahim.H
    Jan 26, 2021 at 12:02
  • 1
    $\begingroup$ @Ibrahim.H When I replace argmax with idxmax, the code works with the new versions of python and libraries. $\endgroup$
    – David Dale
    Jan 26, 2021 at 14:00

As of version 0.24, it does!

Announcement, documentation

  • 1
    $\begingroup$ I wish the outdated answeres could be deleted. I didn't even notice your answer the first couple of times I looked at this thread. $\endgroup$
    – anonuser01
    Apr 26, 2021 at 0:26
  • $\begingroup$ What you suggest is not correct. The forward stepwise selection does not require n_features_to_select to be set beforehand, but the sklearn's sequentialfeatureselector (the thing that you linked) does. So these two thing are different. $\endgroup$
    – Eiffelbear
    Oct 9, 2021 at 8:47
  • 1
    $\begingroup$ @Eiffelbear there's a PR for that. $\endgroup$
    – Ben Reiniger
    Oct 9, 2021 at 12:32

Sklearn DOES have a forward selection algorithm, although it isn't called that in scikit-learn. The feature selection method called F_regression in scikit-learn will sequentially include features that improve the model the most, until there are K features in the model (K is an input).

It starts by regression the labels on each feature individually, and then observing which feature improved the model the most using the F-statistic. Then it incorporates the winning feature into the model. Then it iterates through the remaining features to find the next feature which improves the model the most, again using the F-statistic or F test. It does this until there are K features in the model.

Notice that the remaining features that are correlated to features incorporated into the model will probably not be selected, since they do not correlate with the residuals (although they might correlate well with the labels). This helps guard against multi-collinearity.

  • 1
    $\begingroup$ Be aware, however: stata.com/support/faqs/statistics/stepwise-regression-problems $\endgroup$
    – makansij
    May 30, 2016 at 20:52
  • 1
    $\begingroup$ Do you mean Select K-best? $\endgroup$
    – Nitro
    May 12, 2017 at 16:38
  • $\begingroup$ yes. Also, you should read this: stats.stackexchange.com/questions/204141/…. $\endgroup$
    – makansij
    May 21, 2017 at 21:50
  • 2
    $\begingroup$ That's sort of forward selection. But it's not generic - it is specific to a linear regression model, whereas typically forward selection can work with any model (model agnostic) as is the RFE and can handle classification or regression problems. But I suspect most people are looking for this use case and it's certainly good to mention it here. $\endgroup$
    – Simon
    Sep 9, 2017 at 20:38
  • 4
    $\begingroup$ This is not a STEPWISE selection, because each p-value is calculated for a univariate regression, independently of all the other covariates. $\endgroup$
    – David Dale
    Nov 7, 2017 at 12:17

In fact there is a nice algorithm called "Forward_Select" that uses Statsmodels and allows you to set your own metric (AIC, BIC, Adjusted-R-Squared, or whatever you like) to progressively add a variable to the model. The algorithm can be found in the comments section of this page - scroll down and you'll see it near the bottom of the page.


I would add that the algorithm also has one nice feature: you can apply it to either classification or regression problems! You just have to tell it.

Try it and see for yourself.


Actually sklearn doesn't have a forward selection algorithm, thought a pull request with an implementation of forward feature selection waits in the Scikit-Learn repository since April 2017.

As an alternative, there is forward and one-step-ahead backward selection in mlxtend. You can find it's document in Sequential Feature Selector

  • $\begingroup$ I used it (from mlxtend), works very well and integrates nicely with scikit-learn, recommended! $\endgroup$
    – dolphin
    Jun 24, 2020 at 11:53






from sklearn.linear_model import LinearRegression
from sklearn.datasets import load_boston

boston = load_boston()
X, y = boston.data, boston.target

lr = LinearRegression()

sfs = SFS(lr, 

sfs = sfs.fit(X, y)
fig = plot_sfs(sfs.get_metric_dict(), kind='std_err')

plt.title('Sequential Forward Selection (w. StdErr)')

I developed this repository link. My Stepwise Selection Classes (best subset, forward stepwise, backward stepwise) are compatible to sklearn. You can do Pipeline and GridSearchCV with my Classes.

  • $\begingroup$ aic, bic, adjusted r-squared, mse are all available for model selection. $\endgroup$
    – HE Xin
    Apr 19, 2020 at 17:58
  • $\begingroup$ The best thing is you can use Pipeline and GridSearchCV. $\endgroup$
    – HE Xin
    Apr 19, 2020 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.