7
$\begingroup$

I am learning about different input-vector representations for Neural Networks

One of the alternatives to sparse One-Hot encoded vector is the Multi-Hot encoding.

Do I understand correctly that a traditional binary approach to counting numbers is exactly what the Multi-Hot is? We can imagine a byte as a vector of 8 components, and each entry is either 0 or 1.0

Second part of the question:

I would like to use Multi-Hot encoding to express up to 255 possible input values. If I use the binary-approach, will this be identical to Label-encoding? In a sense that network will figure out that 00000010 is "superior" to 00000001, that there is a "strong correlation and precedence"?

Or is it less exaggerated? For example, in Label-encoding, I could merely use 1 input neuron, just with a varied strength of 0 to 255, much like an enum. There the effect would be really exaggerated, as it presents a really obvious precedence of value $A$ over value $A-1$

This means I can't use it for 255 distinct categories (or for relatively unrelated categories)

Is the effect as bad in Multi-Hot encoding, in particular in a binary approach?


If it's the same as Label-Encoding (which only uses 1 input neuron), why would people ever consider Multi-hot, bloating the dimension of the input-vector?

This post points out we can use multi-hot when the input should contain $N$ concatenated one-hot vectors. For example to represent $N$ entities each of which can belong to $Z$ distinct categories. Is there also another use?

$\endgroup$

2 Answers 2

5
$\begingroup$

You can think of binary encoding as a compromise between label encoding and one-hot encoding. For distinct categories, label encoding introduces a false linear order that brings a lot of noise into the model (category 1 < category 2 < category 3....) . Binary encoding introduces false additive relationships between the categories (e.g. category 4 + category 1 = category 5 or 100 + 001 = 101) but fewer of them.

Therefore, binary will usually work better than label encoding, however only one-hot encoding will usually preserve the full information in the data.

Unless your algorithm (or computing power) is limited in the number of categories it can handle, one-hot encoding will be preferred over other encoding schemes. If you are limited, mean encoding is a powerful alternative because it transform a categorical feature into a numeric one (giving you the minimal number of inputs) while preserving the most important information in the data.

[Mean encoding replaces every category with its target mean. The mean encodings need to be constructed carefully on a separate dataset to avoid data leakage. If you want to reduce your input dimensions as much as possible, this can be helpful. Mean encoding can also be helpful as an additional feature and is very popular on Kaggle to squeeze out some extra performance. This is also known as target encoding or likelihood encoding.]

$\endgroup$
3
  • $\begingroup$ Good answer. I also want to add for future readers to remember that one-hot input vector isn't as wasteful (in execution performance) as it might seem at the first glance. There is no need to pass the input through a proper weight-matrix in first layer. That's because one-hot input vector always has only 1 non-zero entry. Two words: Embedding Matrix. They allow to save speed. It's done by pre-computing the incoming values in all neurons of our first layer, given some one-hot index. Then, we simply take an appropriate row from our matrix, and activate it. (No need for expensive dot-products) $\endgroup$
    – Kari
    Nov 16, 2018 at 17:44
  • $\begingroup$ Also, there is no need to waste memory by having a one-hot vector that has 1000 zeros and just one entry as 1.0; Instead we can just store it as an index. We then can use such an index to get an appropriate entry in the embedding matrix $\endgroup$
    – Kari
    Nov 16, 2018 at 17:46
  • 1
    $\begingroup$ that's correct. sorry, my answer wasn't specific to modern frameworks for neural networks. $\endgroup$
    – oW_
    Nov 16, 2018 at 17:49
0
$\begingroup$

Do I understand correctly that a traditional binary approach to counting numbers is exactly what the Multi-Hot is? We can imagine a byte as a vector of 8 components, and each entry is either 0 or 1.0

I strongly believe so.

This means I can't use it for 255 distinct categories (or for relatively unrelated categories)

Is the effect as bad in Multi-Hot encoding, in particular in a binary approach?

I found a pretty interesting answer. It seems that binary can actually be used with classification tasks really well!
Have a look at the table on the last page of paper "A Comparative Study of Categorical Variable Encoding Techniques for Neural Network Classifiers" 2017

enter image description here

This would mean we can save the number of input neurons BY AN INCREDIBLE AMOUNT, instead of using traditional one-hot encoding.

Personally, I can get it intuitively: Label-encoding tells us to set a neuron with different values: 1,2,3,4... It's really easy for the network to linearly-interpolate from 1 to 2 and from 2 to 3, by using fractions. Thus, there is a really strong precidence between such input values, and the network will easily pick up on that. So we can't use Label-encoding for categories.

Contrary to that, Binary encoding exhibits a more "integer-like" behavior. In other words, it's not as blatantly evident how to linearly interpolate from 1 to 2 in binary form (from 0001 to 0010), which resembles one-hot approach too :)

$\endgroup$
2
  • 1
    $\begingroup$ This study is not very good... all the variables are ordinal and have at most 4 levels (which can be represented in order(!) as 00,01,10,11 from which it is easy to infer the ordinal relationship, however this becomes impossible if you have more many levels). I don't think you can draw any conclusions about encoding schemes for bigger, categorical (not necessarily ordinal) data. For big categorical data with many levels, a NN with binary encoding will not outperform a NN with one-hot encoding if you make it big enough and train it appropriately. $\endgroup$
    – oW_
    Nov 15, 2018 at 23:57
  • $\begingroup$ Ah, good point! :( But I don't hope to outperform one-hot, I merely wish it would be as good as one-hot $\endgroup$
    – Kari
    Nov 16, 2018 at 0:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.