I am trying to use Linear Regression, to predict salary in USD. I have the following data:
Data:
- 607 records
- Numerical columns: year, salary, salary in USD
- Categorical columns: experience, type, residence, currency, remote work, company location, and company size.
- Target: salary in USD
Preprocessing dataset:
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.compose import ColumnTransformer
# Columns to drop:
drop_cols = ['Currency', 'Company location', 'Salary', 'Title']
# Attributes of interest
num_attributes = ['Year']
one_hot_attributes = ['Experience', 'Type', 'Remote work', 'Residence', 'Company size']
# Drop columns:
data.drop(drop_cols, 1, inplace=True)
# Setup transformer for column:
preprocessor = ColumnTransformer([
('nums', StandardScaler(), num_attributes),
('one_hot', OneHotEncoder(drop='first', sparse=False), one_hot_attributes)],
remainder='passthrough')
Pipe:
from sklearn.pipeline import Pipeline
pipe = Pipeline(steps =[
('preprocessor', preprocessor),
('model', LinearRegression()),
])
pipe.fit(X_train, y_train)
Perform prediction:
prediction = pipe.predict(X_test)
pd.DataFrame({'original test set':y_test, 'predictions': prediction})
Error:
ValueError: Found unknown categories ['IR', 'HN', 'MT', 'PH', 'NZ', 'CZ', 'MD'] in column 3 during transform