I’ve been trying to build a NN tokenizer where the inputs would be chars and the outputs, tokens.

But it is not clear to me how this kind of model should work in terms of the output format. If the outputs are tokens, they could be represented as embeddings, one-hot or maybe indices/ints extracted from the embeddings?

Source codes I found doing something similar are either old or not that straightforward for learning.

Some of my questions are:

Can you describe the shape and meaning of the inputs and outputs for such model?

Is it possible to use an embedded output (or the inverse of an embedding layer to output an integer representing a token)?

If the output is one-hot, doesn’t it get too heavy since the total number of tokens would be around 100k to 1m (number of possible english words)?

Are there some tutorials or examples you’d recommend of a tokenizer being trained using keras/tensorflow (hopefully 2.0)?


First, a clarification: normally, we distinguish tokenization and word segmentation. It is not clear to me what you exactly mean. Word segmentation is normally needed in languages without spaces between words, like Chinese. Tokenization is applied to most languages and is the process of splitting words in subword units to obtain a manageable vocabulary size and data sparsity levels.

Second, a word of warning: in normal systems, both word segmentation and tokenization is usually not done with neural networks, but with other approaches, like dictionary-based, CRFs or iterative approaches. Maybe the most popular Chinese word segmenter in Python is Jieba. Most state of the art NLP models use some variant of byte-pair encoding (BPE) tokenization.

Now, the answer. This kind of neural models, normally doesn't generate a sequence of tokens, because defining the set of all possible tokens in advance would defeat the purpose of having a dynamic tokenizer. Instead, your model could tag each letter to mark if it's the beginning of a word, the ending of a word, or the middle of a word, or if it's a single-letter word. This article does precisely that for Chinese word segmentation with a biLSTM.

If you google "neural word segmentation github", you will find several open-source projects for word segmentation. A quick search did not reveal any project with tensorflow 2.0. In my opinion, tensorflow is not the most popular deep learning framework for sequence stuff, so sticking with it may reduce the number of available open-source projects.


Not the answer you're looking for? Browse other questions tagged or ask your own question.