I am working on a character-based Language Generator, loosely based on this tutorial on the TensorFlow 2.0 website. Following the example, I am using an
Embedding() layer to generate character embeddings: I want my model to generate text character by character.
My vocabulary counts 86 unique characters. What embedding size should I choose?
Should I always choose an embedding size that is shorter than the size of vocabulary? The embedding size in the example above is much larger than the vocabulary size, I can't understand how this can build an effective model (but apparently it does, if it's an official tutorial, and if anyone can explain me why it'd be much appreciated).
Another thing I find puzzling is: when we generate word embeddings is because we want a dense representation of a word meaning. Does it make sense to make it larger than the actual one-hot encoded vectors we started with?