In a classification problem when the response variable has multi-class, e.g., "sunny","rainy","cloudy", how should we encode it? I know that for predictors like this, usually we do One Hot Encoding, and if a predictors have too many classes, then we might just use the LabelEncode.

When this multi-class issue occurs in the response variable, I guess we can just use LabelEncode() rather use using One Hot encoding right? Because if we use One Hot encoding, then we will have 2 variables as the response variable, and the machine learning algorithm in sklearn usually expects the response variable not to be a vector right? (I mean it expects a long 1D vector with length equal to the number of observations but not a 2D matrix). But on the other hand, if we just map "sunny", "rainy","cloudy" to {1,2,3} or {0,1,2}, or whatever 3 numbers, that will create a less than or greater than relationship among "sunny", "rainy","cloudy", which is not inherited in the original problem.

  • $\begingroup$ Is this a general classification/prediction question? Please explain what is LabelEncode() and sklearn. If you are talking about some specific library then, either tag or mention. $\endgroup$
    – Ironluca
    Apr 19 '17 at 13:57
  • $\begingroup$ "...and if a predictors have too many classes, then we might just use the LabelEncoder..." that is not the difference between using OneHot vs Label. $\endgroup$
    – gented
    Jan 20 '20 at 10:25

It depends on the meaning of the classes, and whether they have any meaningful order.

If they are ordinal or a scale, then there is a meaningful ordering, and it can potentially be reasonable to order them and then assign labels $1, 2, \dots, n$ in order to the classes. This is what you call "LabelEncode". In some cases, some form of regression might also be appropriate and worth trying.

If they are categorical or nominal, with no meaningful ordering, then LabelEncode makes no sense and will be no better than a one-hot encoding -- and potentially worse. So, the default is one-hot encoding.

If they are categorical but have some hierarchy, there are smarter methods. You can build a tree that represents the hierarchy, then have a multi-hot encoding. If there are $n$ classes, you will have a tree with $n$ leaves and (say) $m$ nodes in total (where $m > n$). Then you can encode the class as a $m$-vector, whose $i$th entry is 1 if the class is a descendant of node $i$ in the tree, or 0 otherwise.

If the classes are categorical but have various attributes, you can try an encoding that has one element for each possible attribute. For instance, if there are $k$ binary attributes, you can have a $k$-vector with a 0 or 1 in the $i$th entry according to whether the corresponding class has the $i$th attribute or not. You can generalize from there for more general situations.

See this and this for more about different types of measurements.


This depends largely on the software. sklearn classifiers will know not to treat label-encoded data as ordered; that said, most/all of them will take the raw string data just fine (and in many cases will use a LabelEncoder internally, for computational efficiency). If you one-hot encode multiclass data, sklearn will generally think your problem is multilabel instead of just multiclass, and your results will be different. See also this SO question.

But, in contrast, most neural network implementations will expect multiclass targets to be one-hot encoded in advance, and presumably you'll want to use a softmax activation on the final layer.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.