I tried to find the answer to this question but I can't find anything, so I ask here: How does Keras Tokenizer choose tokens given a sentence of words ?
To be more precise with what I want to know, given this simple example:
#Import module from keras.preprocessing.text import Tokenizer # define a document doc = ['The cat sat on the mat'] # create the tokenizer tokenizer = Tokenizer() # fit the tokenizer on the document tokenizer.fit_on_texts(doc) encoded_doc=tokenizer.texts_to_sequences(doc) print('word_index : ',tokenizer.word_index)
This method creates the vocabulary index based on word frequency and then it basically takes each word in the text and replaces it with its corresponding integer value from the
Therefore, this means that in the step in which tokenizer is fit on the document (I think in this step), it decides that the tokens are the words of the sentence. Why ? Is it possible to change this choice and choose as tokens the letters of the sentence ?
Thank you in advance.