0
$\begingroup$

I have a bunch of data of employees and their salaries.

I would like to build a regression model that predicts

The columns in question are countries, employment_status, job_title, education

All of these are string values as follows:

data.country.unique():
array(['Slovenia', 'United States', 'Sweden', 'United Kingdom', 'Canada',
       'New Zealand'], dtype=object)
data.employment_status.unique():
array(['Full time', 'Independent or freelancer or company owner',
       'Part time'], dtype=object)
data.job_title.unique()
array(['Developer', 'DBA', 'Other', 'Data Scientist', 'Manager',
       'Architect', 'Analyst', 'Engineer', 'Sales',
       'Analytics consultant', 'Principal database engineer',
       'Sr Consultant'], dtype=object)
data.education.unique():
array(['Bachelors (4 years)', 'Masters', 'Associates (2 years)',
       'None (no degree completed)', 'Doctorate/PhD'], dtype=object)

In order to train this data on a model, here is what I am doing:

from sklearn.tree import DecisionTreeRegressor
train_x = data[['id', 'country', 'employment_status', 'job_title', 'education']]
train_y = data[['salary']]
model = DecisionTreeRegressor()
model.fit(train_x, train_y)

I have just been maping the string values in these colums and converting them to numbers. For example:

# country:
country_map = {
    'Slovenia': 1, 
    'United States': 2, 
    'Sweden': 3, 
    'United Kingdom': 4, 
    'Canada': 5,
    'New Zealand': 6
}
# employment_status:
employment_status_map = {
    'Independent or freelancer or company owner': 0,
    'Part time': 1,
    'Full time': 2
}
# education:
education_map = {
    'Bachelors (4 years)': 2, 
    'Masters': 3, 
    'Associates (2 years)': 1,
    'None (no degree completed)': 0, 
    'Doctorate/PhD': 4
}

My question is is there a better way to do it so that it is more meaningful? and makes more sense? for example someone from US will no doubt make more money than someone from Slovenia. Is there a way to map this using pandas so that this information is captured during training?

$\endgroup$

2 Answers 2

1
$\begingroup$

You can multiple things here :

  1. Converting them to numerical introduces some sense of ordering For example if you say slovenia is 1 and USA is 2 ans ordering is introduced instead you can use one hot encoding.

     Pandas getdummies function will do it for you
    
  2. If one of your string has a lot of values say 1000 one hot encoding does not makes sense. In those cases people use Target encoding or weight of evidence

$\endgroup$
1
$\begingroup$

These string data, called categorical data can be converted to numerical data using many Categorical Encoding Techniques. Encoding categorical data is a process of converting categorical data into integer format so that the data with converted categorical values can be provided to the different models.


Types of Categorical Techniques:

  • Backward Difference Coding
  • BaseN
  • Binary
  • CatBoost Encoder
  • Count Encoder
  • Generalized Linear Mixed Model Encoder
  • Hashing
  • Helmert Coding
  • James-Stein Encoder
  • Leave One Out
  • M-estimate
  • One Hot
  • Ordinal
  • Polynomial Coding
  • Sum Coding
  • Target Encoder
  • Weight of Evidence
  • Wrappers
  • Quantile Encoder
  • Summary Encoder

More details on these encoding techniques can be found in the category_encoders documentation

Useful Links


$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.