The goal of what I'm trying to accomplish here is to have the output contain all of the use_cols but the model only be built to calculate on categorical_features. The output will then be used to predict and compare the prediction 'REVIEW_ACTION' to the actual 'REVIEWER_ACTION'. Ignoring for the moment why this is A BAD THING TO DO, can we focus on how to achieve this?

            'SSO', 'USER_ID', 'ASSET_ID', 'ROLE', 'HPA', 
prediction_col = ['REVIEW_ACTION']

categorical_features = ['DIVISION_NAME', 'PERSON_TYPE', 'JOB_CHANGE', 'JOB_TRANSFER',
                       'SSO', 'USER_ID', 'ASSET_ID', 'ROLE', 'HPA', 'LAST_LOGIN_DATE',
                       'USER ID', 'LABELED_ROLE']

categorical_transformer = Pipeline(steps=[
    ('si', SimpleImputer(strategy='constant', fill_value='missing')),
    ('ohe', OneHotEncoder(handle_unknown='ignore'))],remainder='passthrough')

preprocessor = ColumnTransformer(
        ('cat', categorical_transformer, categorical_features)])

rf = Pipeline(steps=[('preprocessor', preprocessor),
                      ('rfc', RandomForestClassifier(n_estimators=100))

I then fit the data and try to run the prediction

df['rf_prediction'] = df[categorical_features].apply(rf.predict)

and get the error:

IndexingError: ('Too many indexers', 'occurred at index DIVISION_NAME') 

This has something to do with the columns being 'passthrough''d but I'm not sure how to resolve it. I don't want to process some of these columns but want them in the results when I write the file so that I can validate the results.

Somewhere in my debugging I took a step back and am seeing different errors when I call fit. If X = data I see:

ValueError: could not convert string to float: 'Herve'

Note that "Herve" is a field that is passed through (remainder='passthrough') so the model shouldn't be seeing it -- or so I've been led to believe.


1 Answer 1


Removing remainder='passthrough' resolved the problem. I just tacked the prediction onto 'X' and got what I needed since sklearn v20 can deal with pandas dataframes.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.