Encoding for Linear Regression

Question

I have a CSV file with salary information and other columns.

I am trying to transform some of these columns into proper values, for a LinearRegression and a SGDRegressor, or some other. Because, I don't think that the LinearRegression in sklearn can handle the data bits as is.

Data:

• 607 records
• Numerical columns: year, salary, salary in USD
• Categorical columns: experience, type, residence, currency, remote work, company location, and company size.
• Target: salary in USD

Encoding:

# Import neccessary encoder
from sklearn.preprocessing import OneHotEncoder

# Encoding of categorical data
encoder = OneHotEncoder(sparse=False)

# Extract columns
columns = data[['Experience', 'Type', 'Residence', 'Remote work', 'Company location', 'Company size']]

Questions:

• How to group any data within the categories (to avoid duplicates)?
• Is OneHotEncoder the recommended way of doing this?

Answer 1

Comments:

• You should not group any data even if there are duplicates, because this would distort the distribution of the values (features and target).
• OneHotEncoder should be used on the categorical features only. Even with those, mind that values which are too rare should usually be removed or replaced in order to avoid overfitting.
• Some algorithms work better with numerical features scaled.
• Linear regression is unlikely to work well with some complex data in my opinion. Personally I like to try decision tree regression for this kind of mixed dataset.