3
$\begingroup$

I am working on a character-based Language Generator, loosely based on this tutorial on the TensorFlow 2.0 website. Following the example, I am using an Embedding() layer to generate character embeddings: I want my model to generate text character by character.

My vocabulary counts 86 unique characters. What embedding size should I choose?

Should I always choose an embedding size that is shorter than the size of vocabulary? The embedding size in the example above is much larger than the vocabulary size, I can't understand how this can build an effective model (but apparently it does, if it's an official tutorial, and if anyone can explain me why it'd be much appreciated).

EDIT:

Another thing I find puzzling is: when we generate word embeddings is because we want a dense representation of a word meaning. Does it make sense to make it larger than the actual one-hot encoded vectors we started with?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

There is a theoretical lower bound for embedding dimension

I would urge you to read this paper, but the gist of it is dimension could be chosen based on corpus statistics

GLOVE paper discussed embedding, check page 7 for graphs. What I want to say with this reference is that you can treat it as hyperparameter and find your optimal value.

EDIT: Here is my personal/borrowed from google rule of thumb. Embedding vector dimension should be the 4th root of the number of categories is start with that, and then I play around it. Read this toward the end when they explain their embedding. Why COULD (it must not) make sence: What is BOW rather than one hot encoding of your n-grams.

Does it make sence to make it larger? it depends. On one hand you are right if we make it too big we loose the distributed representation property of the word embedding matrix, on the other hand it works in praxis.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Any rule of thumb? What I find puzzling is: when we generate word embedding, we want a dense representation of a word meaning. Does it make sense to make it larger than the actual one-hot encoded vectors we start with? $\endgroup$
    – Leevo
    Commented Dec 20, 2019 at 22:32
  • 1
    $\begingroup$ But the example referenced is text from Shakespeare so the number if character classes can't be no more than 80 (A-Z, a-z and some special characters). The fourth root of that is 3, yet the embedding dimension used in the example is 256. Why 256 and not some other arbitrary number? $\endgroup$
    – Gaslight Deceive Subvert
    Commented May 31, 2020 at 2:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.