One hot encoding at character level with Keras

Question

I am reading Chollet's book on deep learning at the moment and in the NLP chapter he says:

Note that Keras has built-in utilities for doing one-hot encoding of text at the word level or character

I have looked into Keras metods and I cannot find which function he is referring to. keras.utils.to_categorical does not seem to be applicable directly here, as it requires int inputs.

I would like to efficiently encode some text at character level for a small RNN project: what can I use there?

Answer 1

I think that you are looking for the keras Tokenizer with the char_level=True flag:

from keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer(char_level=True)
tokenizer.fit_on_texts(your_dataset_train)
sequence_of_int = tokenizer.texts_to_sequences(your_dataset_train_or_test)

Now that you have sequences of Integer, you can use keras.utils.to_categorical =)