2
$\begingroup$

I have a CSV file with salary information and other columns.

enter image description here

I am trying to transform some of these columns into proper values, for a LinearRegression and a SGDRegressor, or some other. Because, I don't think that the LinearRegression in sklearn can handle the data bits as is.

Data:

  • 607 records
  • Numerical columns: year, salary, salary in USD
  • Categorical columns: experience, type, residence, currency, remote work, company location, and company size.
  • Target: salary in USD

Encoding:

# Import neccessary encoder
from sklearn.preprocessing import OneHotEncoder

# Encoding of categorical data
encoder = OneHotEncoder(sparse=False)

# Extract columns
columns = data[['Experience', 'Type', 'Residence', 'Remote work', 'Company location', 'Company size']]

Questions:

  • How to group any data within the categories (to avoid duplicates)?
  • Is OneHotEncoder the recommended way of doing this?
$\endgroup$
2
  • $\begingroup$ What's your question? Your post gives a lot of background info but it's not clear what you need help with $\endgroup$
    – zachdj
    Oct 12, 2022 at 19:38
  • $\begingroup$ Just an observation: you have missed currency in your extract columns. $\endgroup$
    – Miss.Alpha
    Oct 12, 2022 at 20:38

1 Answer 1

3
$\begingroup$

Comments:

  • You should not group any data even if there are duplicates, because this would distort the distribution of the values (features and target).
  • OneHotEncoder should be used on the categorical features only. Even with those, mind that values which are too rare should usually be removed or replaced in order to avoid overfitting.
  • Some algorithms work better with numerical features scaled.
  • Linear regression is unlikely to work well with some complex data in my opinion. Personally I like to try decision tree regression for this kind of mixed dataset.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.