Full disclosure this question is based on following this tutorial: https://tinyurl.com/vmyj8rf8
I am trying to fully understand embedded layers in Keras. Imagine having a network to try and understand basic sentiment analysis as a binary classifier (1 positive sentiment and 0 negative sentiment). The toy dataset for this is as follows:
# Define 10 restaurant reviews
reviews =[
'Never coming back!',
'horrible service',
'rude waitress',
'cold food',
'horrible food!',
'awesome',
'awesome services!',
'rocks',
'poor work',
'couldn\'t have done better'
]#Define labels
labels = array([1,1,1,1,1,0,0,0,0,0])
This data can be used to train a really simple network as follows:
Vocab_size = 50
model = Sequential()
embedding_layer = Embedding(input_dim=Vocab_size,output_dim=8,input_length=max_length)
model.add(embedding_layer)
model.add(Flatten())
model.add(Dense(1,activation='sigmoid'))
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['acc'])
print(model.summary())
In order to feed this data into he network, we can one hot encode it using Keras one_hot as follows:
encoded_reviews = [one_hot(d,Vocab_size) for d in reviews]
print(f'encoded reviews: {encoded_reviews}')
We get the following output:
encoded reviews: [[14, 45, 43], [8, 2], [6, 43], [24, 1], [8, 1], [11], [11, 21], [16], [34, 40], [2, 25, 36, 15]]
I understand that the purpose of setting Vocab_size = 50, even though there are only around 20 unique words in the corpus is to give a large enough hashing space for the hashing algorithm behind one_hot to avoid collisions when the text is encoded.
If I train the model on these words (assume fixed length input and padding) and then get the weights of the embedded layer:
print(embedding_layer.get_weights()[0].shape)
(50, 8)
We can see this it is an array of 50 vectors that look like this as an example:
[ 0.17051394 0.13659576 -0.05245572 -0.12567708 0.06743167 0.05893507 -0.14506021 0.06448647]
My understanding is that each of these vectors corresponds to a word embedding for each word in the corpus. But if there are only 20 unique words in the corpus and Vocab_size is set larger than this then that can't be completely true? If Vocab_size > corpus_vocab_size, then what do these embeddings represent? Any help would be appreciated.