1
$\begingroup$

I'm wondering why increasing the dimension of a word dimension vector in NLP doesn't necessarily lead to a better result. For instance, on examples I run, I see sometimes that using a pre-trained 100d GloVe vector performs better than a 300d one. Why is this the case? Intuitively a larger dimension should become almost like a one-hot encoding and be in some more "accurate," no?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You can think about phenomenons close to the curse of dimensionality.

Embedding words in a high dimension space requires more data to enforce density and significance of the representation. A good embedding space (when aiming unsupervised semantic learning) is characterized by orthogonal projections of unrelated words and near directions of related ones. For neural models like word2vec, the optimization problem (maximizing the log-likelihood of conditional probabilities of words) might become hard to compute and converge in high dimensional spaces.

You’ll often have to find the right balance between data amount/variety and representation space size.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.