208
votes
Accepted
When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?
There are some cases where LabelEncoder or DictVectorizor are useful, but these are quite limited in my opinion due to ...
- 6,798
55
votes
When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?
While AN6U5 has given a very good answer, I wanted to add a few points for future reference. When considering One Hot Encoding(OHE) and Label Encoding, we must try and understand what model you are ...
- 731
27
votes
K-Means clustering for mixed numeric and categorical data
(In addition to the excellent answer by Tim Goodman)
The choice of k-modes is definitely the way to go for stability of the clustering algorithm used.
The clustering algorithm is free to choose any ...
- 1,313
24
votes
K-Means clustering for mixed numeric and categorical data
This question seems really about representation, and not so much about clustering.
Categorical data is a problem for most algorithms in machine learning. Suppose, for example, you have some ...
- 575
20
votes
How can I appropriately handle cleaning of gender data?
There are at least two general considerations to make:
Domain-related
If an attribute potentially has predictive power in your domain and more specifically for your task your models might benefit ...
- 5,360
17
votes
Accepted
Why do we need to discard one dummy variable?
Simply put because one level of your categorical feature (here location) become the reference group during dummy encoding for regression and is redundant. I am quoting form here "A categorical ...
- 4,207
16
votes
How to convert categorical data to numerical data in Pyspark
This can be done using StringIndexer in PySpark and the reverse using IndexToString for reference please check this:
...
- 1,147
16
votes
Accepted
Confusion about Entity Embeddings of Categorical Variables - Working Example!
For those who are interested, I've spent some time, finally figured out that the problem was the way one has to prepare the categorical encoding for the Entity Embedding suitable for a neural network ...
- 4,207
14
votes
How can I do classification with categorical data which is not fixed?
It is very good question; in fact this problem has been around for a while and I have not yet found the perfect solution. Yet more than happy to share my experience:
Avoid one-hot-encode as much as ...
- 4,207
12
votes
Accepted
Mass convert categorical columns in Pandas (not one-hot encoding)
If your categorical columns are currently character/object you can use something like this to do each one:
...
- 136
11
votes
Clustering for mixed numeric and nominal discrete data
Taking a stab:
I am trying to identify a clustering technique with a similarity measure that would work for categorical and numeric binary data.
Gower Distance is a useful distance metric when the ...
- 111
11
votes
Accepted
Is there an asymmetric version of nominal correlation?
I found what I was looking for - it's called Theil's U, or the Uncertainty Coefficient.
I've used it in this Kaggle kernel, you can check it out for an example and code implementation in Python
EDIT:...
- 689
10
votes
One Hot encoding for large number of values
If you really care about the number of dimensions, you still can try to apply a dimensionality reduction algorithm, such as PCA (Principal Component Analysis) or LDA (Linear Discriminant Analysis), ...
- 619
10
votes
Accepted
How can I dynamically distinguish between categorical data and numerical data?
I'm not aware of a foolproof way to do this. Here's one idea off the top of my head:
Treat values as categorical by default.
Check for various attributes of the data that would imply it is actually ...
- 578
9
votes
How to combine categorical and continuous input features for neural network training
There's three main approaches to solving this:
Building two models separately and then training an ensemble algorithm that receives the output of the two models as an input
Concating all the data ...
- 191
9
votes
How can I appropriately handle cleaning of gender data?
It is quite an interesting question. I guess that you can call it "dealing with non-binary gender roles in a binary language" or something like this.
In the past I did once something similar....
- 6,082
8
votes
How to combine PCA and MCA on mixed data?
You may want to use Factor analysis of mixed data.
It allows you to do dimension reduction on a complete data set.
A R implementation could be found in the FactoMineR package. But this function ...
- 241
8
votes
How to deal with categorical feature of very high cardinality?
This is an old question. I am surprised that I don't see anyone mentioned Mean Encoding (a.k.a Target Encoding). It is very popular in supervised learning problems. Besides, I have seen people use ...
- 181
8
votes
Accepted
Why don't tree ensembles require one-hot-encoding?
The encoding leads to a question of representation and the way that the algorithms cope with the representation.
Let's consider 3 methods of representing n categorial values of a feature:
A single ...
- 2,623
8
votes
Different number of features in train vs test
Even though @Jekaterina Kokatjuhha's answer is accepted, I completely disagree with what it suggests. You should never make use of your test set when creating your pipeline. Technically, you don't ...
- 5,456
8
votes
Accepted
How to handle columns with categorical data and many unique values
For categorical columns, you have two options :
Entity Embeddings
One Hot Vector
For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change ...
- 2,239
8
votes
Why does frequency encoding work?
Check this post.
In the cases where the frequency is related somewhat with the target variable, it helps the model to understand and assign the weight in direct and inverse proportion, depending on ...
- 814
7
votes
K-Means clustering for mixed numeric and categorical data
You should not use k-means clustering on a dataset containing mixed datatypes. Rather, there are a number of clustering algorithms that can appropriately handle mixed datatypes. Some possibilities ...
7
votes
K-Means clustering for mixed numeric and categorical data
It depends on your categorical variable being used. For ordinal variables, say like bad,average and good, it makes sense just to use one variable and have values 0,1,2 and distances make sense here(...
- 349
7
votes
One-Hot Vector representation vs Label Encoding for Categorical Variables
The main difference I can think of is that using one-hot encoding will mean that all your strings will be at the same (hamming) distance from each other, while using a scalar value means that ...
- 330
7
votes
Accepted
How to deal with categorical feature of very high cardinality?
One-hot-encoded ZIP codes shouldn't present a problem with modern tools, where features can be much wider (millions, billions even), but if you really want you could aggregate area codes into regions, ...
- 10.5k
7
votes
Accepted
Why after adding categorical data the Linear Regression fails?
One possible reason is that when you use one-hot-encoding for categorical data, you should set the intercept property in the function to be False:
...
- 1,429
7
votes
Accepted
Pandas categorical variables encoding for regression (one-hot encoding vs dummy encoding)
One advantage of get_dummies is that it can operate on values other than integers (so you don't need the LabelEncoder) and ...
oW_♦
- 6,264
7
votes
How to deal with missing data for only some categories
There are three types of missing data: Missing Completely At Random (MCAR), Missing At Random (MAR) and Missing Not At Random (MNAR).
Your case is the second, where according to wikipedia it:
...
- 311
6
votes
When to use One Hot Encoding vs LabelEncoder vs DictVectorizor?
LabelEncoder is for ordinal data, while OHE is for nominal data.
- 87
Only top scored, non community-wiki answers of a minimum length are eligible