0
$\begingroup$

Background:

I have created a basic modeling workflow in sklearn that utilizes sklearn's pipeline object. There are some preprocessing steps within the pipeline, and the last step of the pipeline is to fit an estimator that performs automatic feature-selection, for example, lasso regression. Thus, I begin my workflow with n features, but ultimately only m (where m < n) of them will be used to make predictions for the final model. A simple example is provided below:

from sklearn.datasets import load_breast_cancer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# Use Wisconsin breast cancer dataset as an example
X, y = load_breast_cancer(return_X_y = True)

# Produce train test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)

# Pipeline preprocesses then applies lasso regression
pipe = Pipeline(steps = [
    ('preprocessing', StandardScaler()),
    ('lasso', LogisticRegression(penalty = 'l1', solver = 'saga', C = 0.4))
]
)

# Fit the pipeline
pipe.fit(X_train, y_train)

# View the coefficients
pipe.named_steps['lasso'].coef_

print('Count of non-zero coefficients: ',
      sum(pipe.named_steps['lasso'].coef_[0] != 0))
print('Count of zero coefficients: ',
      sum(pipe.named_steps['lasso'].coef_[0] == 0))

Count of non-zero coefficients: 20

Count of zero coefficients: 10

Objective:

I am now looking to apply my model to a new set of observations to produce predictions for them, but it seems computationally wasteful to run all of the n features through the pipeline, as only m of them will ultimately contribute to the final prediction. The preprocessing steps are not theoretically required on the other n-m features once the model has been fit. Of course this example uses a toy dataset, but on larger datasets the excessive processing could be material.

# Predictions require all features to exist in the X dataset
pipe.predict_proba(X_test)

# Passing a subset of the features causes an error - and understandably so since the pipeline
# lacks the context needed to determine which features are included
pipe.predict_proba(X_test[:,np.where(pipe.named_steps['lasso'].coef_[0] != 0)[0]])

ValueError: X has 20 features, but StandardScaler is expecting 30 features as input.

Is there a solution that would allow me to 'pare down' an already-fitted pipeline object so that it can operate on only the m features that are theoretically needed to make my predictions going forward?

$\endgroup$
1
  • $\begingroup$ You may use ColumnTransformer with passthrough for the 10 Features. $\endgroup$ – 10xAI Feb 3 at 16:49
1
$\begingroup$

Standard Scalar trained on 30 features so it expects 30 features only. One simple hack you can do is, you can create a new Standard Scalar and train with those 20 features, and replace your pipeline Standard Scalar with the new one.

For the LogisticRegression, get the non zero weights and set those weights to the new model with 20 features without any training like below.

model.coef_ = numpy_array_of_non_zero_coef
model.intercept_ = numpy_array_of_intercept
$\endgroup$
1
  • 1
    $\begingroup$ This fits my purpose very well. I've also discovered that LogisticRegression also has an attribute called n_features_in_ that is likely worth updating. Also, an alternative to fitting a new StandardScaler, is to update the mean_, var_, scale_, and n_features_ values, in a similar manner to your suggestion w/ handling LogisticRegression. Thanks! $\endgroup$ – DataScienceRick Feb 4 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.