2
$\begingroup$

I am reading Chollet's book on deep learning at the moment and in the NLP chapter he says:

Note that Keras has built-in utilities for doing one-hot encoding of text at the word level or character

I have looked into Keras metods and I cannot find which function he is referring to. keras.utils.to_categorical does not seem to be applicable directly here, as it requires int inputs.

I would like to efficiently encode some text at character level for a small RNN project: what can I use there?

$\endgroup$
5
$\begingroup$

I think that you are looking for the keras Tokenizer with the char_level=True flag:

from keras.preprocessing.text import Tokenizer
tokenizer = Tokenizer(char_level=True)
tokenizer.fit_on_texts(your_dataset_train)
sequence_of_int = tokenizer.texts_to_sequences(your_dataset_train_or_test)

Now that you have sequences of Integer, you can use keras.utils.to_categorical =)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.