4
$\begingroup$

I've been working on a NN-based classification system that accepts document vectors as input. I can't really talk about what I'm specifically training the neural net on, so i'm hoping for a more general answer.

Up to now, the word vectors I've been using (specifically, the gloVe function from the text2vec package for R) have been target vectors. Up to now I wasn't aware that the word2vec training produced context vectors, and quite frankly I'm not sure what exactly they represent. (It's not part of the main question, but if anybody could point me to resources on what context vectors are for and what they do, that would be greatly appreciated)

My question is, how useful are these context word vectors in any kind of classification scheme? Am I missing out on useful information to feed into the neuralnet?

How would, qualitatively speaking, these four schemes fare?

  1. Target word vectors only.
  2. Context word vectors only.
  3. Averaged target and context vectors.
  4. Concatenated vectors (i.e. a 100-vector word2vec model ends up with a length of 200)
$\endgroup$
3
  • $\begingroup$ Thank you for your question, before I give an answer, could you clarify what you mean by how word2vec would fair on the schemes? Do you mean how “well” will they represent the above, for example? $\endgroup$
    – shepan6
    Jul 21 '20 at 6:20
  • $\begingroup$ I don't really have the words to accurately explain what I mean... I guess how well they would pave the way for a NN to discriminate meaningfully between different classes of text. Keep in mind I don't really know what the context vectors are for in the first place. A qualitative handwavey explanation will suffice. $\endgroup$
    – Ingolifs
    Jul 21 '20 at 6:54
  • $\begingroup$ Hi @Ingolifs, it would be good if you can accept any reply that answers your question to help others that might have the same question. $\endgroup$
    – Carl
    Jul 26 at 9:51
0
$\begingroup$

Essentially the target vectors $\mathbf{w}$ and the context vectors $\mathbf{c}$ learn the same thing (they jointly factorise a matrix of word co-occurrence statistics and are two sides of the same coin). You can use either equivalently.

However, a better result is given by taking their average... This is because the embeddings are trained so that $\mathbf{w}^\top\mathbf{c}$ approximates a useful statistic. If the $\mathbf{c}$'s are discarded and only $\mathbf{w}$'s used then $\mathbf{w}^\top\mathbf{w}$ gets back the encoded statistic but with an error. Using $\mathbf{w}^\top\mathbf{c}$ is obviously preferable as it gets back the statistic with minimal error, but requires keeping both sets of embeddings (twice the memory). Under some simplifying assumptions, using their average $\mathbf{a}=\tfrac{\mathbf{w}+\mathbf{c}}{2}$ can be seen to half the error, so not as good as keeping both (e.g. concatenating), but better than just using either $\mathbf{w}$ or $\mathbf{c}$.

More detail is given in this paper

$\endgroup$
-1
$\begingroup$

So, from what I understand from the question, you want to get an idea of how word2vec works so you can assess how well the resulting context vectors from this model will help discriminate between words by their meaning.

Word2vec works on the premise of the distributional hypothesis which essentially states that words which appear in soimikar contexts will have similar meanings (e.g. the dog ate the food/ the cat ate the food : both dog and cat appear in the same context so they are semantically close to each other)

So word2vec formulates this by a CBOW model which effectively is a feedforward neural network, which takes in the surrounding context of the target word as a series of one hot encoded vectors and aims to predict the target word (there is an assumption made of course where the contexts are treated as a bag of words and therefore assumes that a word’s meaning is not related to the word ordering of its context).

After training this, the models weights are then used to dorm the word embeddings, which represents a words meaning in semantic space. (For further reference)

$\endgroup$
-1
$\begingroup$

I am a bit confused about your question because word2vec basically outputs a word vector/embedding for each word which is completely independent of the context of that word in the document.

Word2vec is a very simple and nice embedding technique Google came up with that uses a neural network to make word embeddings by using either a Skip-gram or Continuous Bag of Words approach (CBOW). If you would like more details you should read their paper.

To overcome this issue with word2vec - that its embeddings do not learn context - there are two approaches. One, a model named ELMO (it uses LSTMs) and the other BERT (uses Transformers). These are the state of the art now as they can learn context and long-range dependencies. I don't have experience with R so I do not know if these can be implemented in R or if there is a package for them, but they work with most deep learning packages like PyTorch or TensorFlow.

A brief summary:

$\endgroup$
1
  • $\begingroup$ Each word2vec embedding of a target word is entirely DEPENDENT on (in fact defined by) the context words found around that target word within the training corpus. This is different to ELMOs/BERT, which adapt a the embedding of a target word at each instance it occurs according to its local context words. $\endgroup$
    – Carl
    Jul 26 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.