ValueError: Found unknown categories ['IR', 'HN', 'MT', 'PH', 'NZ', 'CZ', 'MD'] in column 3 during transform

Question

I am trying to use Linear Regression, to predict salary in USD. I have the following data:

Data:

• 607 records
• Numerical columns: year, salary, salary in USD
• Categorical columns: experience, type, residence, currency, remote work, company location, and company size.
• Target: salary in USD

Preprocessing dataset:

from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.compose import ColumnTransformer

# Columns to drop:
drop_cols = ['Currency', 'Company location', 'Salary', 'Title']

# Attributes of interest
num_attributes = ['Year']
one_hot_attributes = ['Experience', 'Type', 'Remote work', 'Residence', 'Company size']

# Drop columns:
data.drop(drop_cols, 1, inplace=True)

# Setup transformer for column:
preprocessor = ColumnTransformer([
    ('nums', StandardScaler(), num_attributes),
    ('one_hot', OneHotEncoder(drop='first', sparse=False), one_hot_attributes)], 
                                remainder='passthrough')

Pipe:

from sklearn.pipeline import Pipeline

pipe = Pipeline(steps =[
    ('preprocessor', preprocessor),
    ('model', LinearRegression()),
])

pipe.fit(X_train, y_train)

Perform prediction:

prediction = pipe.predict(X_test)

pd.DataFrame({'original test set':y_test, 'predictions': prediction})

Error:

ValueError: Found unknown categories ['IR', 'HN', 'MT', 'PH', 'NZ', 'CZ', 'MD'] in column 3 during transform

Answer 1

This error is thrown by the OneHotEncoder class because your test dataset contains values for a column (likely the Residence column) that were not present in your training dataset. As specified in the documentation, the default for the handle_unknown argument is to throw an error when new values are encountered when transform is called. Setting handle_unknown='ignore' should stop the error from being thrown.