Does One-Hot encoding increase the dimensionality and sparsity of dataset?

Question

There are two ways to convert object datatype into numeric datatype, first is One-Hot encoding and second is simply map the numerical tags to different values.

For example for column Age containing three distinct values 'child', 'adult' and 'old', for that column One-Hot encoding is:

Age	Age_child	Age_adult	Age_old
child	1	0	0
adult	0	1	0
old	0	0	1

Whereas a simple mapping of numerical tags to distinct values might be

Age	_Age
child	1
adult	2
old	3

What I understand One-Hot encoding can increase the number of columns many times. For instance, consider 10 columns and each column having 3 distinct values on average, then the resulting dateset will have 30 columns. Whereas, simple numerical mapping does not change the datasets size (columns) and simply assigns the numerical tags to each distinct value.

So the question is, does One-Hot encoding increase the dimensionality and sparsity of complex and large dataset? What is the more appropriate approach for machine or deep learning analyses out of these two? Is there any pros and cons of both?

Answer 1

Which encoding technique to use depends on your data/features. Ordinal encoding is used when there ia a sense of order in your feature. For example you have a feature performance which has values worst, bad good. Here you should use ordinal encoder which will result in worst = 0, bad = 1 and good = 2. We used ordinal encoding because good is better than bad which is better than worst. So here we have a sense of order with good getting more priority. The model will then learn this sense of order.

OHE is used when there is no sense of order present and we simply just want to convert categorical type to numerical type. For example we have a feature named color which has values as red, blue and green. If we use ordinal encoding, it will assign red = 0, blue = 1 and green = 2 which will mean green is much more important than blue and red. But this makes no sense!

Hence in the second case it would be wise to use OHE.

Coming to the pros and cons, yes OHE increases the dimensionality of the dataset and ordinal encoding does not. But OHE is useful when there is no ordering in the feature.

So yes it depends on the feature type. I would suggest you to use both wherever necessary. For example you have some features where there is no sense of order and other features where order is present. Use both techniques so as to remain true to the feature type and also reduce the dimensionality a bit!!

Answer 2

What you describe is ordinal encoding. If there is an inherent order to your data (such as age), you can definitely try it. And yes, one-hot encoding does increase dimensionality and sparsity of the data. But these two are not the only ways to handle categorical data. Here is a list of different encoders, and a paper where some of these techniques are compared using neural networks.

Answer 3

You are correct - one hot encoding, by definition, increases your dimensions and (most likely) also the sparsity.

Your numerical mapping can be rather misleading since e.g a random forest would interpret adult>child which, in the case of age, makes sense. But say the mapping was {1:"dog",2:"cat",3:"horse"}and not the age of it then 2>1 does not makes sense.

When to use what depends on what task you try to solve, and what algorithm you want to use