3
$\begingroup$

I have one categorical variable of string type in my dataset. I need to convert it to numerical value for further processing. I know standard way to represent categorical data is to use one-hot encoding. But that will convert each entry of the variable to a vector.

LabelEncoder of sklearn converts each entry to a scalar value. I realise this is a very naive and possibly stupid question but which representation is more commonly used and is there a reason for the bias?

Improve this question
$\endgroup$
2
  • $\begingroup$ What kind of "further processing" are you planning to do? That should determine what representation fits best. For classification and regression, there are methods that can deal with categorical variables inherently, for example decision trees. $\endgroup$
    – Alexander Bauer
    May 28, 2016 at 9:58
  • $\begingroup$ datascience.stackexchange.com/questions/9443/… $\endgroup$
    – Vikas Gupta
    Dec 25, 2017 at 15:36

2 Answers 2

Reset to default
7
$\begingroup$

The main difference I can think of is that using one-hot encoding will mean that all your strings will be at the same (hamming) distance from each other, while using a scalar value means that distances between the resulting features will be meaningless (it may encode "red" as 1, "blue" as 2 and "green" as 3, but there is no reason why red is more similar to blue than to green).

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Yes, that makes sense. So how are these features handled in real life? One-hot vector would increase number of features which could affect the performance (due to overfitting and all). $\endgroup$
    – SHASHANK GUPTA
    Jan 13, 2016 at 16:02
  • 1
    $\begingroup$ But the alternative would create a feature that adds misleading information. Using a one-hot encoding and then performing some form of unsupervised feature selection (based on frequency) to keep your number of features lower would probably be a better solution. If you are worried about overfitting, use a regularized learner. You may also be interested in the accepted answer of this SO question. $\endgroup$
    – Jérémie Clos
    Jan 13, 2016 at 16:32
  • $\begingroup$ I suggest one-hot followed by some dimensionality reduction like PCA. PCA will find linear correlations in the binary features and eliminate them. Don't worry too much about overfitting apriori. Instead wait until you actually see some overfitting and then reduce the dimensionality or add some regularization. $\endgroup$
    – AN6U5
    Jan 18, 2016 at 0:59
2
$\begingroup$

When to use label encoding versus one-hot encoding.

Tree based methods:

When categorical feature is ordinal label encoding can lead to better quality if it preserves correct order of values. In this case a split made by a tree will divide the feature to values 'lower' and 'higher' that the value chosen for this split.

Non-tree based methods:

One-hot encoding or embedings should be used. Unless there is a linear relashionship between the label encoding and the dependent variable non-tree based methods will have a hard time with label encoding.

One-hot encoding a categorical feature with huge number of values can lead to high memory consumption. You can use sparse matrices to deal with this problem. You can also ignore a subset of the categories that are rare to decrease the number of new features.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.