Understand the interpretability of word embeddings

Question

When reading the Tensorflow tutorial for Word Embeddings, I found two notes that confuse me:

Note: Experimentally, you may be able to produce more interpretable embeddings by using a simpler model. Try deleting the Dense(16) layer, retraining the model, and visualizing the embeddings again.

And also:

Note: Typically, a much larger dataset is needed to train more interpretable word embeddings. This tutorial uses a small IMDb dataset for the purpose of demonstration.

I don't know exactly what it means by "more interpretable" in those two notes, is it related to the result displayed by the embedding projector? And also why the interpretability will increase when reducing model's complexity?

Many thanks!

Answer 1

"Interpretable" is not very precise in this context.

In the case of deleting a dense layer, the embedding layer is more likely can learn the nontask dependent co-occurrences of words in the dataset.

In the second case of adding more data, the embedding layer would learn more signals because there is an increased opportunity to "average out" the noise.

In other words, word embeddings are more generalizable by reducing the complexity of the architecture and training on more data.