How to handle non ordinal Features like Gender,Language,Region etc? Ordinal Encoding or one-hot encoding?

Question

I see that usually, while preparing the dataset. Usually, data scientists convert non-ordinal features like Gender or Language in a dataset using LabelEncoder/ordinalEncoder. Ideally, they should have done One-hot encoding right? Won't introducing ordinality affect the models by bringing unwanted bias?

Answer 1

One hot encoder is your best choice. Still, you have to deal with enlarged dimensional size, though, as long as you don't drop categories you are not giving preferences. Dropping is needed to avoid collinearity which is a fancy way to say: "I have four friends, Anne, Bart, Carrie and Dylan. One of them is here with me. It is not Anne, it is not Carrie and it is not Dylan." Collinearity would be created if I would blatantly say that "it is Bart". Some models cannot handle collinearity.

A way to reduce factors and have a meaningful dropping would be to reduce the dimensions needed for your model to a minimum by using approaches as PCA. If you have a linear regression to perform, you can use L1, L2 or both (also called Ridge, Lasso and elastic) regression. Though, you will have to find a metric that informs you when enough dimensions are taken into account.

Answer 2

Any numerical encoding necessarily introduces some ordering even where there is none, simply because numbers have order, whatever they may mean for us.

Even one-hot-encoding introduces order since $1$ is greater than $0$, right?

So any numerical encoding introduces order. One-hot might seem better, but if you ponder on the fact that it enlarges the dimensionality of the problem a lot, and one can suffer from the curse of dimensionality (which is another serious problem), along with the artificially introduced ordering, then it might not be better at all.

One-hot (or one-cold) encoding still has its uses (various architectures may provide better results with one-hot/one-cold), but it is not the undisputed go-to method regarding categorical variables. Hope this is clear by now.

UPDATE: As per @BenReiniger's comment I quote from When to use One Hot Encoding vs LabelEncoder vs DictVectorizor? on plausible criteria for choosing one encoding method over another:

We apply OHE when:

1. When the values that are close to each other in the label encoding correspond to target values that aren't close (non - linear data).
2. When the categorical feature is not ordinal (dog,cat,mouse).

We apply Label encoding when:

1. The categorical feature is ordinal (Jr. kg, Sr. kg, Primary school, high school ,etc).
2. When we can come up with a label encoder that assigns close labels to similar categories: This leads to less splits in the tress hence reducing the execution time.
3. When the number of categorical features in the dataset is huge: One-hot encoding a categorical feature with huge number of values can lead to (1) high memory consumption and (2) the case when non-categorical features are rarely used by model. You can deal with the 1st case if you employ sparse matrices. The 2nd case can occur if you build a tree using only a subset of features. For example, if you have 9 numeric features and 1 categorical with 100 unique values and you one-hot-encoded that categorical feature, you will get 109 features. If a tree is built with only a subset of features, initial 9 numeric features will rarely be used. In this case, you can increase the parameter controlling size of this subset. In xgboost it is called colsample_bytree, in sklearn's Random Forest max_features.