1
$\begingroup$

I was taking an online ML course and the lecturer said that a rule of thumb for choosing the number of dimensions when embedding categorical data is the following

embedding vector dimension should be the 4th root of the number of categories

The lecturer worked for Google and when I looked on the internet for this I only found a Google blog which quickly mentioned it google blog link. I'm guessing it's something that they came up with at Google but was wondering if somebody else has maybe seen it in a research paper.

$\endgroup$

1 Answer 1

0
$\begingroup$

google has published a word embedding that has embedding dimension of 300. Following the rule you have given it should have trained on $300^4 = 8.1*10^9$ words. If google is using ngrams instead of just words, then it seems plausible.

$\endgroup$
4
  • $\begingroup$ Do you mean Google might be using words + n-grams? $\endgroup$
    – zipline86
    Jun 24, 2020 at 8:58
  • $\begingroup$ Maybe frequent ngrams, i.e., well known phrases. $\endgroup$ Jun 24, 2020 at 9:45
  • $\begingroup$ 4th root, not to the power of 4 - as in, a square root of a square root. $\endgroup$
    – NL23codes
    Apr 2, 2022 at 8:27
  • 1
    $\begingroup$ @NL23codes Roots are the inverse of powers. There is nothing wrong here. $\endgroup$
    – Brady Gilg
    Nov 25, 2022 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.