From the documentation of TextVectorization:

max_tokens: Maximum size of the vocabulary for this layer. This should only be specified when adapting a vocabulary or when setting pad_to_max_tokens=True. Note that this vocabulary contains 1 OOV token, so the effective number of tokens is (max_tokens - 1 - (1 if output_mode == "int" else 0)).

output_mode="int": Outputs integer indices, one integer index per split string token. When output_mode == "int", 0 is reserved for masked locations; this reduces the vocab size to max_tokens - 2 instead of max_tokens - 1

The above quotes explain that in the typical encoding scenario (output_mode==int), there are two reserved tokens (OOV and masking), so the actual tokens (are num_tokens-2).

Question 1: It is mentioned that masking is encoded as integer 0, so to which integer is OOV encoded?

Question 2: In the following popular tutorial from keras.io (https://keras.io/examples/nlp/text_classification_from_scratch/):

vectorize_layer = keras.layers.TextVectorization(
text_input = keras.Input(shape=(1,), dtype=tf.string, name='text')
x = vectorize_layer(text_input)
x = layers.Embedding(max_features + 1, embedding_dim)(x)

Can someone explain why +1 is added in the embedding? The TextVectorization has already accounted for the two reserved keywords. Also, assuming the TextVectorization can only generate |max_features| different outputs from (0 to max_features-1), it seems impossible to generate |max_features + 1| as requested in the embedding parameter.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.