0
$\begingroup$

I have searched a lot for this issue but unfortunately came up with nothing. Usually in a ML model, during preprocessing, we use Pipelines and ColumnTransformer to group together preprocessing steps and the algorithm. Now the problem with Pipelines is that it performs the specified preprocessing for all the columns. For example if I specify:-

pipeline = Pipeline(steps = [('scale', StandardScaler()), ('encode', OneHotEncoder())])

The above pipeline will apply Standard scaler to all the columns of the dataset and the encoder will apply to all categorical column. I don't want that. I want different preprocessing techniques for different columns. For example out of 5 numerical columns, I want Standard scaling only for 3 columns and for the rest 2 I want Robust scaling. Similarly for out of 4 categorical columns, I want OneHotEncoder for 2 columns and LabelEncoder for 2 columns.

How can I implement this using Pipelines or make_pipelines? Or can it even be implemented?

Edit based on comment:

I tried the ColumnTransofrmer method but it gives me an error on fitting:

data = pd.read_csv('cars_sampled.csv')
data

data1 = data.copy(deep = True)
data1

y = np.log(data2['price'])
data2.drop(['price'], axis = 1, inplace = True)

num1 = ['powerPS']
num2 = ['kilometer', 'age']
cat_impute1 = ['fuelType', 'gearbox', 'model']
cat_impute2 = ['vehicleType', 'notRepairedDamage']
cat_encode1 = ['model', 'fuelType', 'gearbox', 'notRepairedDamage', 'vehicleType', 'brand']


preproc = ColumnTransformer(transformers = [                                            
                                            ('num1', StandardScaler(), num1), 
                                            ('num2', RobustScaler(), num2),
                                            
                            ('cat_impute1', SimpleImputer(strategy = 'most_frequent'), cat_impute1), 
                            ('cat_impute2', SimpleImputer(strategy = 'constant', fill_value = 'Missing'), cat_impute2),

                                            
                                            ('cat_encode1', OneHotEncoder(), cat_encode1) 
                                            ])

pipeline = Pipeline(
    steps=[
        ("preprocessor", preproc),
        ("model", LinearRegression())
    ]
)

train_x, test_x, train_y, test_y = train_test_split(data2, y, test_size = 0.25, random_state = 69)

pipeline.fit(train_x, train_y)

The error is in the last line pipeline.fit(train_x, train_y) and it is as follows:

ValueError: For a sparse output, all columns should be a numeric or convertible to a numeric.

Edit 2:

data = pd.read_csv('cars_sampled.csv')
data

data1 = data.copy(deep = True)
data1

y = np.log(data2['price'])
data2.drop(['price'], axis = 1, inplace = True)

num1 = ['powerPS']
num2 = ['kilometer', 'age']
cat_impute1 = ['fuelType', 'gearbox', 'model']
cat_impute2 = ['vehicleType', 'notRepairedDamage']
cat_encode1 = ['model', 'fuelType', 'gearbox', 'notRepairedDamage', 'vehicleType', 'brand']

cat_pipe = Pipeline(steps = [('impute', SimpleImputer(strategy = 'most_frequent')),
                ('encode', OneHotEncoder())])


preproc = ColumnTransformer(transformers = [                                            
                                            ('num1', StandardScaler(), num1), 
                                            ('num2', RobustScaler(), num2),
                                            ('cat', cat_pipe, cat_encode1) 
                                            ])

pipeline = Pipeline(
    steps=[
        ("preprocessor", preproc),
        ("model", LinearRegression())
    ]
)

train_x, test_x, train_y, test_y = train_test_split(data2, y, test_size = 0.25, random_state = 69)

pipeline.fit(train_x, train_y)
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You mentioned the ColumnTransformer, which you should be able to use to achieve this (see also this page from the scikit-learn documentation:

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, RobustScaler, OneHotEncoder, LabelEncoder
from sklearn.linear_model import LogisticRegression

preprocessor = ColumnTransformer(
    transformers=[
        ("num1", StandardScaler(), ["col1", "col2", "col3"]),
        ("num2", RobustScaler(), ["col4", "col5"]),
        ("cat1", OneHotEncoder(), ["col6", "col7"]),
        ("cat2", LabelEncoder(), ["col8", "col9"]),
    ]
)

pipeline = Pipeline(
    steps=[
        ("preprocessor", preprocessor),
        ("classifier", LogisticRegression())
    ]
)
Improve this answer
$\endgroup$
14
  • $\begingroup$ The page from scikit learn does not have any code like the one you mention in your answer where we can apply different preprocessing steps for different columns by mentioning them as a list in ColumnTransformer. $\endgroup$
    – spectre
    Nov 30, 2021 at 9:10
  • $\begingroup$ It does, see the second code block in the provided link. They have specified two types of preprocessing steps for either numerical or categorial features. The specific columns to apply the transformers to are specified by the variables numeric_features and categorical_features, which are both lists. $\endgroup$
    – Oxbowerce
    Nov 30, 2021 at 9:14
  • $\begingroup$ Unfortunately the above method does not work. See my updated question. $\endgroup$
    – spectre
    Nov 30, 2021 at 9:46
  • $\begingroup$ That error is not caused by the way you are using the ColumnTransformer, but by the fact that you are using the LabelEncoder on feature columns instead of the target labels. The documentation explicitly says 'This transformer should be used to encode target values, i.e. y, and not the input X.'. See also the source code, where the fit_transform method only takes two arguments (self and y). $\endgroup$
    – Oxbowerce
    Nov 30, 2021 at 10:05
  • 1
    $\begingroup$ This seems to be caused by the fact that the pipeline is trying to convert the data array to a sparse array, for which all values need to be numbers. This is however not possible as you still have some values which are strings (i.e. the categorical columns) since you are applying the imputing and encoding steps in parallel (as separate transforms in the ColumnTransformer) instead of sequentially. $\endgroup$
    – Oxbowerce
    Nov 30, 2021 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.