1
$\begingroup$

I create a tokenizer with

import tf
tokenizer = tf.keras.preprocessing.text.Tokenizer(split='', char_level=True, ...)
tokenizer.fit_to_texts(...)

But when I convert sequences of tokens to texts, the result contains a space after each character (except for the last one):

test_text = 'this is a test'
seq = tokenizer.texts_to_sequences([test_text])
r = tokenizer.sequences_to_texts(seq)[0]
assert(r == ''.join([ c+' ' for c in test_text ])[:-1])

Is there a way to avoid this added spaces? Am I missing some configuration parameter?

$\endgroup$

1 Answer 1

0
$\begingroup$

This is a consequence of the (erroneous?) working of character level tokenizer in Keras.

A simple way to correct the output is to delete every second character in the output string:

seq_no_spaces = [text[::2] for text in seq]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.