6
$\begingroup$

Using Embedding layer in Keras on a fairly small vocabulary (~300), I am looking at how to choose the output of this layer (dense vector) when given a 300 dimension vector. I think that the embedded vector need to have a minimum length to be able to map a given vocabulary.

$\endgroup$
7
$\begingroup$

The ratio of vocabulary vs embedding length to determine the size of other layers in a neural network doesn't really matter. Word embeddings are always around 100 and 300 in length, longer embedding vectors don't add enough information and smaller ones don't represent the semantics well enough. What matters more is the network architecture, the algorithm(s) and the dataset size.

A simple way to understand this concept is that a bidirectional LSTM model with 50 neurons (nodes) followed by a fully connected layer of 70 neurons will outperform a simple MLP of 1000 neurons (nodes) connected to a embedding layer simply due to its architecture. Adding dropout will improve performance as well.

In addition, even if the vocabulary is just 300 words, using pre-trained embeddings will probably yield better results than training the embeddings directly on the dataset. The same applies to data size, a dataset with more samples will make a better classifier than a dataset with just a couple thousand samples.

In summary, it is preferable to try many architectures and cross-validate them (and/or ensemble them depending if you have a large enough dataset) with the smallest number of neurons possible and then start building up in size, depending on what computational resources you have and the speed of development you need. Large models slow down development speed whereas small models speed it up. This goes whether your vocabulary is the size of common crawl or just 300. As usual, try feature engineering (sentence length, special characters, etc.) and increase the dataset size as doing so often helps in whatever task you're trying to predict.

| improve this answer | |
$\endgroup$
3
$\begingroup$

A similar question was asked here.

This Google Developers blog post says:

Well, the following "formula" provides a general rule of thumb about the number of embedding dimensions:

embedding_dimensions =  number_of_categories**0.25

That is, the embedding vector dimension should be the 4th root of the number of categories.

Interestingly, the Word2vec Wikipedia article says (emphasis mine):

Nevertheless, for skip-gram models trained in medium size corpora, with 50 dimensions, a window size of 15 and 10 negative samples seems to be a good parameter setting.

Assuming a standard-ish sized vocabulary of 1.5 million words, this rule of thumb comes surprisingly close:

50 == 1.5e6 ** 0.2751

| improve this answer | |
$\endgroup$
  • $\begingroup$ So for OP's question of a 300 word vocabulary, the embedding size should be.. 4? According to this rule of thumb? $\endgroup$ – Shamoon May 23 '19 at 14:47
  • $\begingroup$ 4.16... round as you choose ;) This a generic answer and probably based on English language vocabs... Maybe it doesn't scale too well on such tiny vocab sizes as 300. Like everything, would be a case of try and see in this specific case, but I feel it's useful information in the general case. $\endgroup$ – Tom Hale May 24 '19 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.