1
$\begingroup$

all,

i'm trying to classify a set of features to belong to a particular company (my dependent variable). my independent variables are a mixture of continuous and categorical features.

my data-set i am training on is labelled data with the label being the company ( the dependent var). I am not sure how i should go about dealing with my dependent variable should i use one hot encoding on the the entire data-set and then split it into training and test?

i am uncertain whether to onehot encode before i feed into decsion tree.

| improve this question | |
$\endgroup$
  • $\begingroup$ Do you have many categories, with respect to the rest of the dataset made of non-categorical variables? $\endgroup$ – Leevo Apr 6 at 13:28
  • $\begingroup$ of my indepdent variables about 80% of columns are categotical. my dependent variable has atleast 400 categories.. so i am not keen to use one hot encoding.. $\endgroup$ – Maths12 Apr 6 at 13:40
  • $\begingroup$ Tree-based models could work better than others with dummy variables. As an alternative, did you try some dimensionality reduction? $\endgroup$ – Leevo Apr 6 at 13:44
  • $\begingroup$ no i may do that, but do i change the dependent variable to dummy after or before i split into train and test? will the dependnet variable not have more than one column - is this allowed? $\endgroup$ – Maths12 Apr 6 at 13:54
1
$\begingroup$

This is called multiclass classification, and the encoding needed for the target variable depends on what package and model you're using. You may be expected to one-hot encode (e.g. neural networks usually have an output neuron for each class), ordinal encode (e.g. most(?) sklearn multiclass classifiers), or leave them as strings (most R models, I'd guess?).

| improve this answer | |
$\endgroup$
  • $\begingroup$ most my data is actually nomial so to one hot encode it all i run into issues. i have reduced the dataset as much as i can but am still left with many nominal columns . i am using decision tree classifier $\endgroup$ – Maths12 Apr 7 at 16:45
  • $\begingroup$ can i just assingn my dependetn variable to be e.g. supermarket_a=1, supermarket_b=2.. etc... would that be a way to handle it? $\endgroup$ – Maths12 Apr 7 at 17:03
  • 1
    $\begingroup$ @Maths12 "supermarket_a=1, supermarket_b=2.. etc" is what I mention above as "ordinal encoding", and whether you should do that or not depends on your package. (The model ought to treat them as unordered, but different packages may deal with this in different ways.) As for the independent variables, I feel that's already answered well elsewhere, but summary: some packages can deal with categoricals internally, others you'll have to one-hot encode or try something like ordinal encoding, target encoding, hashing, binary encoding, clustering, .... $\endgroup$ – Ben Reiniger Apr 7 at 20:23
0
$\begingroup$

Theoretically - Decision tree doesn't need any encoding for Categorical data. You just train, cross val and it will work.

But Library implementation mandates to encode in Numerical which is better to manage instead of a mix of Char and Number.

What that means is - We just have to encode our feature in some consistent form. We need not worry about LabelEncoder, Standardizing.

These operations are needed when the Model uses some Mathematics dependent algorithm to optimize e.g Gradient Descent (then 10 will be treated as 2 times 5 and will result in an error). But this is not the case with DecisionTree

My code on Iris. Using 3 ways to encode y.

  1. As it is (0,1,2) - Comment 2, 3 to run

  2. A random(Consistent) choice - Comment 3 to run

  3. OHE - Comment 2 to rum

from sklearn import datasets
import numpy as np, pandas as pd
iris = datasets.load_iris()
X = iris.data  

###Case 1 - Target as 0,1,2
y = iris.target
###Case 2 - Target as 31,41,71
#y = pd.Series(iris.target).map({0:31,1:41,2:71})

###Case 3 - Target as OneHot
from sklearn.preprocessing import OneHotEncoder
ohe = OneHotEncoder(sparse=False)
y = ohe.fit_transform(y.reshape(-1,1))

from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(X,y,test_size=0.25)
from sklearn.tree import DecisionTreeClassifier
model = DecisionTreeClassifier(max_depth=3)
model.fit(x_train, y_train)
model.score(x_test, y_test), model.predict(x_test)
| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ This depends on the implementation; e.g., sklearn (currently) requires encoded categorical features. $\endgroup$ – Ben Reiniger Apr 8 at 18:17
  • $\begingroup$ i thought the sklearn version needed it to be encoded $\endgroup$ – Maths12 Apr 9 at 9:14
  • $\begingroup$ Updated. Please see if it helps. My previous answer was not detailed and also it didn't reflect what I wanted to convey $\endgroup$ – 10xAI Apr 9 at 9:49
  • $\begingroup$ As for encoding the target, in sklearn's DecisionTreeClassifier, the first two approaches are identical, but the third is considered as a multilabel problem. (Since iris is rather easy, I still get the same results on the test set, but on e.g. the digits dataset you can get samples with predict returning all zeros, i.e. none of the digits scored highly enough to get predicted.) $\endgroup$ – Ben Reiniger Apr 9 at 15:06
  • $\begingroup$ I assumed user is just looking for conceptual knowledge. I believe these are all multi-class scenario. It would be multi-label when we have both Versicolor and Virginica in one image. I mean both probabilities are independent. In case of multi-class both are dependent i.e. if one is 0.6 other can't be > 0.4. gombru.github.io/2018/05/23/cross_entropy_loss @ben-reiniger $\endgroup$ – 10xAI Apr 9 at 17:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.