I have one categorical variable of string type in my dataset. I need to convert it to numerical value for further processing. I know standard way to represent categorical data is to use one-hot encoding. But that will convert each entry of the variable to a vector.

LabelEncoder of sklearn converts each entry to a scalar value. I realise this is a very naive and possibly stupid question but which representation is more commonly used and is there a reason for the bias?

  • $\begingroup$ What kind of "further processing" are you planning to do? That should determine what representation fits best. For classification and regression, there are methods that can deal with categorical variables inherently, for example decision trees. $\endgroup$ – Alexander Bauer May 28 '16 at 9:58
  • $\begingroup$ datascience.stackexchange.com/questions/9443/… $\endgroup$ – Vikas Gupta Dec 25 '17 at 15:36

The main difference I can think of is that using one-hot encoding will mean that all your strings will be at the same (hamming) distance from each other, while using a scalar value means that distances between the resulting features will be meaningless (it may encode "red" as 1, "blue" as 2 and "green" as 3, but there is no reason why red is more similar to blue than to green).

  • 1
    $\begingroup$ Yes, that makes sense. So how are these features handled in real life? One-hot vector would increase number of features which could affect the performance (due to overfitting and all). $\endgroup$ – SHASHANK GUPTA Jan 13 '16 at 16:02
  • 1
    $\begingroup$ But the alternative would create a feature that adds misleading information. Using a one-hot encoding and then performing some form of unsupervised feature selection (based on frequency) to keep your number of features lower would probably be a better solution. If you are worried about overfitting, use a regularized learner. You may also be interested in the accepted answer of this SO question. $\endgroup$ – Jérémie Clos Jan 13 '16 at 16:32
  • $\begingroup$ I suggest one-hot followed by some dimensionality reduction like PCA. PCA will find linear correlations in the binary features and eliminate them. Don't worry too much about overfitting apriori. Instead wait until you actually see some overfitting and then reduce the dimensionality or add some regularization. $\endgroup$ – AN6U5 Jan 18 '16 at 0:59

When to use label encoding versus one-hot encoding.

Tree based methods:

When categorical feature is ordinal label encoding can lead to better quality if it preserves correct order of values. In this case a split made by a tree will divide the feature to values 'lower' and 'higher' that the value chosen for this split.

Non-tree based methods:

One-hot encoding or embedings should be used. Unless there is a linear relashionship between the label encoding and the dependent variable non-tree based methods will have a hard time with label encoding.

One-hot encoding a categorical feature with huge number of values can lead to high memory consumption. You can use sparse matrices to deal with this problem. You can also ignore a subset of the categories that are rare to decrease the number of new features.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.