Scikit-learn pipeline with scaling, dimensionality reduction, average prediction of multiple regression models, and grid search cross validation

Question

I would like to use a sklearn pipeline doing this :

( - ) scale the data ( StandardScaler )

( - ) reduce dimensionality ( PCA )

( - ) make a prediction with GradientBoostingRegressor() and GridSearchCV() ( to get the model with best parameters from grid )

( - ) make a prediction with RandomForestRegressor() and GridSearchCV() ( to get the model with best parameters from grid )

( - ) take the mean of both predictions

but I cannot figure out how to proceed.

Do I have to ( scale & predict ) 2 times or can I scale and then predict 2 times ?

Thks.

Answer 1

You might looking for sklearn.ensemble.VotingRegressor which takes the mean of two regression models.

Here is an example to get you started:

from sklearn.datasets        import make_regression
from sklearn.decomposition   import PCA
from sklearn.ensemble        import GradientBoostingRegressor, RandomForestRegressor, VotingRegressor
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline        import Pipeline
from sklearn.preprocessing   import StandardScaler

# Make fake data     
X, y = make_regression(n_samples=1_000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=42)

pipe = Pipeline([('scl', StandardScaler()),
                 ('pca', PCA()),
                 ('vr', VotingRegressor([('gbr', GradientBoostingRegressor()), ('rfr', RandomForestRegressor())]))
                ])

search_space = [{'vr__gbr__learning_rate':    [.07, .1, .15]}]

gs_cv = GridSearchCV(estimator=pipe,
                     param_grid=search_space,
                     n_jobs=-1)

gs_cv.fit(X_train, y_train)
gs_cv.predict(X_test)

Answer 2

The problem seems to be that sklearn pipelines are strictly linear in nature, so you can't fit the two models within the pipeline. You could try ensembling the GBM and RF into a new model object to use; for that, use FeatureUnion:
https://stats.stackexchange.com/questions/139042/ensemble-of-different-kinds-of-regressors-using-scikit-learn-or-any-other-pytho
https://stackoverflow.com/questions/43010914/stack-ensemble-estimators-using-sklearn-pipeline-and-gridsearchcv
I don't see immediately how to take the simple average of the scores rather than fitting a linear model, but it shouldn't be hard; maybe as a custom transformation? Then there's also a minor difference from what you're describing: the cross-validation for hyperparameters would happen for the ensemble, so would seek the best result of the ensemble rather than the two individual models.