0
$\begingroup$

I am trying to use Linear Regression, to predict salary in USD. I have the following data:

enter image description here

Data:

  • 607 records
  • Numerical columns: year, salary, salary in USD
  • Categorical columns: experience, type, residence, currency, remote work, company location, and company size.
  • Target: salary in USD

Preprocessing dataset:

from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.compose import ColumnTransformer

# Columns to drop:
drop_cols = ['Currency', 'Company location', 'Salary', 'Title']

# Attributes of interest
num_attributes = ['Year']
one_hot_attributes = ['Experience', 'Type', 'Remote work', 'Residence', 'Company size']

# Drop columns:
data.drop(drop_cols, 1, inplace=True)

# Setup transformer for column:
preprocessor = ColumnTransformer([
    ('nums', StandardScaler(), num_attributes),
    ('one_hot', OneHotEncoder(drop='first', sparse=False), one_hot_attributes)], 
                                remainder='passthrough')

Pipe:

from sklearn.pipeline import Pipeline

pipe = Pipeline(steps =[
    ('preprocessor', preprocessor),
    ('model', LinearRegression()),
])

pipe.fit(X_train, y_train)

Perform prediction:

prediction = pipe.predict(X_test)

pd.DataFrame({'original test set':y_test, 'predictions': prediction})

Error:

ValueError: Found unknown categories ['IR', 'HN', 'MT', 'PH', 'NZ', 'CZ', 'MD'] in column 3 during transform
$\endgroup$

1 Answer 1

2
$\begingroup$

This error is thrown by the OneHotEncoder class because your test dataset contains values for a column (likely the Residence column) that were not present in your training dataset. As specified in the documentation, the default for the handle_unknown argument is to throw an error when new values are encountered when transform is called. Setting handle_unknown='ignore' should stop the error from being thrown.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.