Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
55 votes

When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?

While AN6U5 has given a very good answer, I wanted to add a few points for future reference. When considering One Hot Encoding(OHE) and Label Encoding, we must try and understand what model you are ...
Pushkaraj Joshi's user avatar
20 votes

How can I appropriately handle cleaning of gender data?

There are at least two general considerations to make: Domain-related If an attribute potentially has predictive power in your domain and more specifically for your task your models might benefit ...
Jonathan's user avatar
  • 5,410
18 votes
Accepted

Why do we need to discard one dummy variable?

Simply put because one level of your categorical feature (here location) become the reference group during dummy encoding for regression and is redundant. I am quoting form here "A categorical ...
TwinPenguins's user avatar
  • 4,279
16 votes

How to convert categorical data to numerical data in Pyspark

This can be done using StringIndexer in PySpark and the reverse using IndexToString for reference please check this: ...
krishna Prasad's user avatar
16 votes
Accepted

Confusion about Entity Embeddings of Categorical Variables - Working Example!

For those who are interested, I've spent some time, finally figured out that the problem was the way one has to prepare the categorical encoding for the Entity Embedding suitable for a neural network ...
TwinPenguins's user avatar
  • 4,279
14 votes

How can I do classification with categorical data which is not fixed?

It is very good question; in fact this problem has been around for a while and I have not yet found the perfect solution. Yet more than happy to share my experience: Avoid one-hot-encode as much as ...
TwinPenguins's user avatar
  • 4,279
12 votes
Accepted

Mass convert categorical columns in Pandas (not one-hot encoding)

If your categorical columns are currently character/object you can use something like this to do each one: ...
george_w_kush's user avatar
11 votes

Clustering for mixed numeric and nominal discrete data

Taking a stab: I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data. Gower Distance is a useful distance metric when the ...
gregorymatchado's user avatar
11 votes
Accepted

Is there an asymmetric version of nominal correlation?

I found what I was looking for - it's called Theil's U, or the Uncertainty Coefficient. I've used it in this Kaggle kernel, you can check it out for an example and code implementation in Python EDIT:...
shakedzy's user avatar
  • 699
11 votes

How to combine categorical and continuous input features for neural network training

There's three main approaches to solving this: Building two models separately and then training an ensemble algorithm that receives the output of the two models as an input Concating all the data ...
Tadej Magajna's user avatar
9 votes

How can I appropriately handle cleaning of gender data?

It is quite an interesting question. I guess that you can call it "dealing with non-binary gender roles in a binary language" or something like this. In the past I did once something similar....
Carlos Mougan's user avatar
8 votes

K-Means clustering for mixed numeric and categorical data

You should not use k-means clustering on a dataset containing mixed datatypes. Rather, there are a number of clustering algorithms that can appropriately handle mixed datatypes. Some possibilities ...
Sam  - Founder of AceAINow.com's user avatar
8 votes

How to deal with categorical feature of very high cardinality?

This is an old question. I am surprised that I don't see anyone mentioned Mean Encoding (a.k.a Target Encoding). It is very popular in supervised learning problems. Besides, I have seen people use ...
Diansheng's user avatar
  • 181
8 votes
Accepted

Why don't tree ensembles require one-hot-encoding?

The encoding leads to a question of representation and the way that the algorithms cope with the representation. Let's consider 3 methods of representing n categorial values of a feature: A single ...
DaL's user avatar
  • 2,633
8 votes

Different number of features in train vs test

Even though @Jekaterina Kokatjuhha's answer is accepted, I completely disagree with what it suggests. You should never make use of your test set when creating your pipeline. Technically, you don't ...
Valentin Calomme's user avatar
8 votes
Accepted

How to handle columns with categorical data and many unique values

For categorical columns, you have two options : Entity Embeddings One Hot Vector For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change ...
Shamit Verma's user avatar
  • 2,259
8 votes

Why does frequency encoding work?

Check this post. In the cases where the frequency is related somewhat with the target variable, it helps the model to understand and assign the weight in direct and inverse proportion, depending on ...
Piotr Rarus's user avatar
7 votes
Accepted

How to deal with categorical feature of very high cardinality?

One-hot-encoded ZIP codes shouldn't present a problem with modern tools, where features can be much wider (millions, billions even), but if you really want you could aggregate area codes into regions, ...
Emre's user avatar
  • 10.5k
7 votes

K-Means clustering for mixed numeric and categorical data

It depends on your categorical variable being used. For ordinal variables, say like bad,average and good, it makes sense just to use one variable and have values 0,1,2 and distances make sense here(...
Ram's user avatar
  • 349
7 votes
Accepted

Why after adding categorical data the Linear Regression fails?

One possible reason is that when you use one-hot-encoding for categorical data, you should set the intercept property in the function to be False: ...
Hobbes's user avatar
  • 1,449
7 votes
Accepted

Pandas categorical variables encoding for regression (one-hot encoding vs dummy encoding)

One advantage of get_dummies is that it can operate on values other than integers (so you don't need the LabelEncoder) and ...
oW_'s user avatar
  • 6,357
7 votes

How to deal with missing data for only some categories

There are three types of missing data: Missing Completely At Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR). Your case is the second, where according to wikipedia it: ...
Mnng's user avatar
  • 311
7 votes

How does a Neural Net handle an unseen class for a Categorical Feature?

Typically feature X would be presented to the NN via one hot encoding. So we have three (boolean) indicator variables, denoting the presence of A, B, or C. If the problem domain admits of D or E ...
J_H's user avatar
  • 1,100
6 votes

When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?

LabelEncoder is for ordinal data, while OHE is for nominal data.
vochicong's user avatar
6 votes
Accepted

Do categorical features always need to be encoded?

You have partly answered this question yourself ("because converting to integers implies that there is an ordering between features"). I will just clarify the terminology a bit more. Categorical ...
hssay's user avatar
  • 1,998
6 votes

Mass convert categorical columns in Pandas (not one-hot encoding)

First, let's create a mcve to play with: ...
Julien Marrec's user avatar
6 votes

Data scaling before PCA: how to deal with categorical values?

You can not use PCA, or at least it is not recommended, for mixed data. It is best to use Factor analysis of mixed data. You are lucky that Prince is a Python package that covers all data scenarios, ...
TwinPenguins's user avatar
  • 4,279
6 votes
Accepted

How can I deal with tiny categories?

It totally depends on the target task and the importance of the "tiny class" for this task: In some tasks with multiple classes where there is no particular emphasis on any specific class, &...
Erwan's user avatar
  • 25.5k
5 votes

Why don't tree ensembles require one-hot-encoding?

xgboost usually performs better after one-hot encoding. Otherwise, it will treat your categorical variables as numerical variables. But most other tree packages ...
Ricardo Cruz's user avatar
  • 3,420
5 votes
Accepted

Feature importance with high-cardinality categorical features for regression (numerical depdendent variable)

It depends on how you're one-hot encoding them. Many automated solutions for that will name all the converted booleans with a pattern so that a categorical variable called "letter" with values A-Z ...
CalZ's user avatar
  • 1,663

Only top scored, non community-wiki answers of a minimum length are eligible