What happens when the vocab size of an embedded layer is larger than the text corpus used in training?

Question

Full disclosure this question is based on following this tutorial: https://tinyurl.com/vmyj8rf8

I am trying to fully understand embedded layers in Keras. Imagine having a network to try and understand basic sentiment analysis as a binary classifier (1 positive sentiment and 0 negative sentiment). The toy dataset for this is as follows:

# Define 10 restaurant reviews
reviews =[
          'Never coming back!',
          'horrible service',
          'rude waitress',
          'cold food',
          'horrible food!',
          'awesome',
          'awesome services!',
          'rocks',
          'poor work',
          'couldn\'t have done better'
]#Define labels
labels = array([1,1,1,1,1,0,0,0,0,0])

This data can be used to train a really simple network as follows:

Vocab_size = 50
model = Sequential()
embedding_layer = Embedding(input_dim=Vocab_size,output_dim=8,input_length=max_length)
model.add(embedding_layer)
model.add(Flatten())
model.add(Dense(1,activation='sigmoid'))
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['acc'])
print(model.summary())

In order to feed this data into he network, we can one hot encode it using Keras one_hot as follows:

encoded_reviews = [one_hot(d,Vocab_size) for d in reviews]
print(f'encoded reviews: {encoded_reviews}')

We get the following output:

encoded reviews: [[14, 45, 43], [8, 2], [6, 43], [24, 1], [8, 1], [11], [11, 21], [16], [34, 40], [2, 25, 36, 15]]

I understand that the purpose of setting Vocab_size = 50, even though there are only around 20 unique words in the corpus is to give a large enough hashing space for the hashing algorithm behind one_hot to avoid collisions when the text is encoded.

If I train the model on these words (assume fixed length input and padding) and then get the weights of the embedded layer:

print(embedding_layer.get_weights()[0].shape)

(50, 8)

We can see this it is an array of 50 vectors that look like this as an example:

[ 0.17051394 0.13659576 -0.05245572 -0.12567708 0.06743167 0.05893507 -0.14506021 0.06448647]

My understanding is that each of these vectors corresponds to a word embedding for each word in the corpus. But if there are only 20 unique words in the corpus and Vocab_size is set larger than this then that can't be completely true? If Vocab_size > corpus_vocab_size, then what do these embeddings represent? Any help would be appreciated.

Answer 1

tf.keras.layers.Embedding(..., embeddings_initializer="uniform"*,..., *kwargs)

• All the weights are initialized with the init strategy
• All learn the optimum values with the backprop
• Weights for which there is no input will have zero output every time, hence no learning.
• Hence these extra weights will remain at their initialization value

You may check these extra weights before and after.

weight = model.layers[0].get_weights() # Save before training
history = model.fit(x, y)

# These two should be same
weight[0][-1]  # Last weight - Before
model.layers[0].get_weights()[0][-1] # Last weight - After