In the following example the text:

'The quick brown fox jumped over the lazy dog.'

gets the encodings [5, 9, 8, 7, 9, 1, 5, 3, 8].

My code:

enter image description here

In this way the words 'brown' and 'dog' get both the encodings of 8 and 'quick' and 'jumped' 9.

What is the point of encoding if unicity is not guaranteed?

https://machinelearningmastery.com/prepare-text-data-deep-learning-keras/ https://keras.io/preprocessing/text/

  • 2
    $\begingroup$ Just a note - formatting code as markdown as opposed to png is always easier for people to read/answer on this site. $\endgroup$ – Ethan Sep 23 at 14:48

It seems to be bad choice of function names since the one_hot function actually is using hashing and is equivalent to hashing_trick (with standard settings).

See this issue: https://github.com/keras-team/keras/pull/6887

When I do:

from keras.preprocessing.text import one_hot
from keras.preprocessing.text import hashing_trick

text = 'The quick brown fox jumped over the lazy dog.'
print(one_hot(text, n=9))
print(hashing_trick(text, n=9))

I get identical results:

[5, 6, 2, 6, 5, 8, 5, 4, 1]
[5, 6, 2, 6, 5, 8, 5, 4, 1]

So in case you want to have "real" (unique) one-hot encoded results, you need to resort to some other solution (there is no Keras built-in solution for this in the moment as I believe).


You could think of it as dimensionality reduction, though it would be important to know how and why it put certain things into buckets. I remember reading about Vowpal Wabbit doing a sophisticated version of this ("feature hashing") in order to handle a "terafeature".



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.