I am doing an essay on the mathematics behind a text classifier with NLP and neural networks and I would like to know how exactly the TOKENIZER function of Keras works. Whether cosine similarity is involved and how the dictionary creation is carried out taking frequency into account. If anyone knows the answer or a book/article where it is reflected, I will be eternally grateful.

MAX_NB_WORDS = 50000
tokenizer = Tokenizer(num_words=MAX_NB_WORDS, filters='!"#$%&()*+,-./:;<=>?@[\]^_`{|}~', lower=True)
tokenizer.fit_on_texts(data['Consumer complaint narrative'].values)

1 Answer 1


Class Tokenizer is used to turn text into a sequence of integers (each integer being the index of a token in a dictionary). The size of the token dictionary is defined when invoking the constructor, with parameter num_words.

To build the token dictionary, you usually invoke fit_on_texts. Tokenizer then takes the text, splits it on the occurrence of the space character (or the one you provide in the split parameter when constructing it), assuming the tokens are separated by blanks. Tokenizer builds its token dictionary by keeping the num_words - 1 most frequent tokens, and assigns an integer value to each possible token.

Then, you can use Tokenizer to turn text into a sequence of integer numbers, which are indexes to tokens. This is done with texts_to_sequences. It just splits tokens on the space character and looks up the token indexes in its internal token dictionary, returning the sequence of those indexes.

After that, you can use other elements to use those indices in your neural network (e.g. Embedding).

The documentation contains all of these pieces of information, along with other functionality supported by this class.

As you can see, there is little math involved here, just frequency counts. No cosine similarity or anything alike.

  • $\begingroup$ Thank you so much. As for embedding, what are the mathematics used? I thought that cosine similarity anda hyperplanes were involved un the vectorización process. $\endgroup$
    – sara
    Commented Dec 7, 2022 at 18:14
  • $\begingroup$ Embedding is just a collection of vectors, basically a backpropagation-friendly vector lookup table. In the forward pass, it receives integers as input and outputs the vectors associated to those indexes. In the backward pass, it updates its vectors. $\endgroup$
    – noe
    Commented Dec 7, 2022 at 18:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.