4
$\begingroup$

Full disclosure this question is based on following this tutorial: https://tinyurl.com/vmyj8rf8

I am trying to fully understand embedded layers in Keras. Imagine having a network to try and understand basic sentiment analysis as a binary classifier (1 positive sentiment and 0 negative sentiment). The toy dataset for this is as follows:

# Define 10 restaurant reviews
reviews =[
          'Never coming back!',
          'horrible service',
          'rude waitress',
          'cold food',
          'horrible food!',
          'awesome',
          'awesome services!',
          'rocks',
          'poor work',
          'couldn\'t have done better'
]#Define labels
labels = array([1,1,1,1,1,0,0,0,0,0])

This data can be used to train a really simple network as follows:

Vocab_size = 50
model = Sequential()
embedding_layer = Embedding(input_dim=Vocab_size,output_dim=8,input_length=max_length)
model.add(embedding_layer)
model.add(Flatten())
model.add(Dense(1,activation='sigmoid'))
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['acc'])
print(model.summary())

In order to feed this data into he network, we can one hot encode it using Keras one_hot as follows:

encoded_reviews = [one_hot(d,Vocab_size) for d in reviews]
print(f'encoded reviews: {encoded_reviews}')

We get the following output:

encoded reviews: [[14, 45, 43], [8, 2], [6, 43], [24, 1], [8, 1], [11], [11, 21], [16], [34, 40], [2, 25, 36, 15]]

I understand that the purpose of setting Vocab_size = 50, even though there are only around 20 unique words in the corpus is to give a large enough hashing space for the hashing algorithm behind one_hot to avoid collisions when the text is encoded.

If I train the model on these words (assume fixed length input and padding) and then get the weights of the embedded layer:

print(embedding_layer.get_weights()[0].shape)

(50, 8)

We can see this it is an array of 50 vectors that look like this as an example:

[ 0.17051394 0.13659576 -0.05245572 -0.12567708 0.06743167 0.05893507 -0.14506021 0.06448647]

My understanding is that each of these vectors corresponds to a word embedding for each word in the corpus. But if there are only 20 unique words in the corpus and Vocab_size is set larger than this then that can't be completely true? If Vocab_size > corpus_vocab_size, then what do these embeddings represent? Any help would be appreciated.

$\endgroup$

1 Answer 1

3
$\begingroup$

tf.keras.layers.Embedding(..., embeddings_initializer="uniform"*,..., *kwargs)

  • All the weights are initialized with the init strategy
  • All learn the optimum values with the backprop
  • Weights for which there is no input will have zero output every time, hence no learning.
  • Hence these extra weights will remain at their initialization value


You may check these extra weights before and after.

weight = model.layers[0].get_weights() # Save before training
history = model.fit(x, y)

# These two should be same
weight[0][-1]  # Last weight - Before
model.layers[0].get_weights()[0][-1] # Last weight - After
$\endgroup$
2
  • $\begingroup$ Amazing, thanks for clearing that up for me! $\endgroup$
    – Sandy Lee
    May 17, 2021 at 8:41
  • $\begingroup$ What is the advantage of using extra weights ? $\endgroup$ Nov 15, 2021 at 8:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.