New answers tagged categorical-data
1
The answer depends on the library you are using. If you are using scikit - you wouldn't need to one hot encode the targets. Scikit handles it automatically.
If you were using keras to build a neural network, you might want to use one hot encoded labels because the built in loss function in keras (e.g categorical crossentropy) expects labels to be one hot ...
1
Suppose you have a question like: "How does car weight affect miles per gallon (mpg)?"
Load and plot the "car data". In the first plot you can clearly see that there is a (more or less) linear relation between $weight$ and $mpg$.
Now you can ask: is there a difference in time? You can add a "dummy" for the years $\leq$ 1975 to ...
1
We just encode the categorical variable to some sort of numerical representation (like one-hot encoding)
The choice of representation matters, because it has to preserve the properties of a categorical variable: one-hot-encoding is a standard option, but directly encoding categorical values as integers is a mistake because it introduces order where there is ...
0
This has something to do with Mutli-colinearity in case if Multiple Linear Regression. Beacause, Keeping k dummies for k levels of a categorical variable is good idea, but there is a redundancy of one level, which is here in separate column. This is not needed since one of the combination will be uniquely representing this redundant column. Hence, its better ...
1
To solve the high cardinality problem, follow the recommendation of Erwan. To solve the second issue you are mentioning, fit the OneHotEncoder on the train set and use the transformer on the test set, like this:
from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown = 'ignore')
enc.fit(train)
X_train = enc.transform(train)
...
0
It's a mistake to use LabelEncoding for a categorical feature, it should be used only for a categorical target variable. This is because it converts values to integers, hence introducing an arbitrary order over the values.
It's not the values which don't appear in the training set which cause the poor performance (you can check), it's very likely because ...
Top 50 recent answers are included
