6
$\begingroup$

I'm using GloVe pre-trained word vectors (glove.6b.50d.txt, glove.6b.300d.txt) as word embedding.

I have a conceptual question:

  • What is the difference between these files?
  • On the other hand, what does the dimension represent in the GloVe pre-trained word vectors?
$\endgroup$
2
  • $\begingroup$ I'm voting to close this question as off-topic because the community is not for questions which try to ask the difference between files. StackOverflow could be a better option. $\endgroup$ Commented Oct 14, 2019 at 6:30
  • $\begingroup$ @ShubhamPanchal What about the second question? $\endgroup$ Commented Oct 14, 2019 at 6:51

2 Answers 2

6
$\begingroup$

Glove creates word vectors that capture meaning in vector space by taking global count statistics. The training objective of GloVe is to learn word vectors such that their dot product equals the logarithm of the words probability of co-occurrence. while optimizing this, you can use any number of hidden representations for word vector.

In the original paper, they trained with 25, 50, 100, 200, 300. These dimensions are not interpretable. after training, we are getting a vector with 'd' dim that captures many properties of that word. If the dimension is increasing, the vector can capture much more information but computational complexity will also increase.

$\endgroup$
3
  • $\begingroup$ Thanks for the valued response. $\endgroup$ Commented Oct 14, 2019 at 10:37
  • $\begingroup$ What does "hey trained with 25, 50, 100, 200, 300" mean? What is 50 ... ? $\endgroup$
    – abdoulsn
    Commented Jun 1, 2020 at 0:21
  • $\begingroup$ @abdoulsn there the numbers represent the size of the vectors in the vocabulary. In a 50 dimensional vocabulary tokens are represented as 50-dimensional vectors, in a 100-dimensional vocabulary tokens are represented as 100-dimensional vectors, etc. $\endgroup$
    – Andy
    Commented Aug 27, 2020 at 19:43
1
$\begingroup$

Good questions!

What is the difference between these files?

Each of these files contains a different set of pre-trained word-embeddings. Both files can be thought of as dictionaries that map words to vectors of length D, where D is 50/300 in your respective files. The only difference between the files is that they contain different length word vectors.

So your two files are essentially equivalent to this:

word_embedding_50_dims  = {<words>: <array of length 50>}

word_embedding_300_dims = {<words>: <array of length 300>}

Which we can generalize to word-embeddings of length D as being:

word_embedding_D_dims   = {<words>: <array of length D>}

what does the dimension represent in the GloVe pre-trained word vectors?

As another answer pointed out, the dimension has no special meaning, it is a hyperparameter, chosen by the creators of GloVe.

Note that this is not saying that the values were not chosen with a purpose, rather that it does not make sense to think of hyper-parameters as being interpretable in the typical sense of the word.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.