I'm working on a sample project and one of the features is the job description of a person (categorical, for example: blue-collar, retired, unknown, unemployed, student, etc.). Since in the future more job descriptions could be used, I don't think one hot encoding it's the best approach. How would you encode it without using one hot encoding??

  • 1
    $\begingroup$ What's the goal of the project? If it's a sample project and you already have a dataset in hand, why worry about additional values in the future? $\endgroup$
    – zachdj
    Mar 23 '20 at 16:49

I would first start by using something like scikit-learn LabelEncoder. Examples are in the documentation.

EDITED : With this option, you convert every string to a integer starting from 1.

You can also think about trying to convert the strings into hash values.

  • 2
    $\begingroup$ LabelEncoder outputs values between 1 and k, not 0 and 1. The problem with this is that by giving some class a greater value than other class, the model might be more biased towards the bigger value, which is what I'm trying to avoid. $\endgroup$
    – shulito
    Apr 29 '19 at 13:29
  • $\begingroup$ The problem with LabelEncoder pointed by @shulito is true. Apart from that, an algorithm could learn that the sum of two categories are the same as a third category, which would make absolutely no sense. I always take LabelEncoding with great care! It looks like a simple and efficient approach, but it may be dangerous! $\endgroup$
    – 89f3a1c
    Oct 23 '19 at 22:38
  • $\begingroup$ Yes my bad, after a few months, I don't know why I wrote 0 to 1. I will edit the answer. You are right, that you might have a biased model but I did have nice surprises in the past trying it anyway. $\endgroup$
    – eetuko
    Oct 25 '19 at 13:46

1) Try with LabelEncoding(fast way, but not best way)

#Assuming that you have data in Pandas dataframe

def encode_to_num_df(df):
        from sklearn.preprocessing import LabelEncoder
        df = df.apply(LabelEncoder().fit_transform)
        return df

2) Custom encoding each label - best way, but the hardest. Detailed explanation there. Thanks to @Djib2011

my_mapping = {'bad': 1, 'worse': 2, 'worst': 3}

df['feature'] = df['feature'].map(my_mapping)



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.