What is the math behind the Keras Tokenizer() function?

Question

I am doing an essay on the mathematics behind a text classifier with NLP and neural networks and I would like to know how exactly the TOKENIZER function of Keras works. Whether cosine similarity is involved and how the dictionary creation is carried out taking frequency into account. If anyone knows the answer or a book/article where it is reflected, I will be eternally grateful.

MAX_NB_WORDS = 50000
MAX_SEQUENCE_LENGTH = 250
tokenizer = Tokenizer(num_words=MAX_NB_WORDS, filters='!"#$%&()*+,-./:;<=>?@[\]^_`{|}~', lower=True)
tokenizer.fit_on_texts(data['Consumer complaint narrative'].values)

Answer 1

Class Tokenizer is used to turn text into a sequence of integers (each integer being the index of a token in a dictionary). The size of the token dictionary is defined when invoking the constructor, with parameter num_words.

To build the token dictionary, you usually invoke fit_on_texts. Tokenizer then takes the text, splits it on the occurrence of the space character (or the one you provide in the split parameter when constructing it), assuming the tokens are separated by blanks. Tokenizer builds its token dictionary by keeping the num_words - 1 most frequent tokens, and assigns an integer value to each possible token.

Then, you can use Tokenizer to turn text into a sequence of integer numbers, which are indexes to tokens. This is done with texts_to_sequences. It just splits tokens on the space character and looks up the token indexes in its internal token dictionary, returning the sequence of those indexes.

After that, you can use other elements to use those indices in your neural network (e.g. Embedding).

The documentation contains all of these pieces of information, along with other functionality supported by this class.

As you can see, there is little math involved here, just frequency counts. No cosine similarity or anything alike.