10
$\begingroup$

I'm a beginner in neural networks and currently I'm exploring the word2vec model. However I'm having a tough time to understand what the feature matrix exactly is.

enter image description here

I can understand that the first matrix is a one-hot encoding vector for a given word, but what does the second matrix signify? More specifically, what does each of those values (ie. 17, 24, 1 etc) mean?

$\endgroup$
  • $\begingroup$ I have a question about the W genereted by google, can you give me more information about the values gived by google? what's this features used ? thx for your help. $\endgroup$ – hambi Nov 23 '17 at 10:44
8
$\begingroup$

The idea behind word2vec is to represent words by a vector of real numbers of dimension d. Therefore the second matrix is the representation of those words.

The i-th line of this matrix is the vector representation of the i-th word.

Let's say that in your example you have 5 words : ["Lion", "Cat", "Dog", "Horse", "Mouse"], then the first vector [0,0,0,1,0] means you're considering the word "Horse" and so the representation of "Horse" is [10, 12, 19]. Similarly, [17, 24, 1] is the representation of the word "Lion".

To my knowledege, there are no "human meaning" specifically to each of the numbers in these representations. One number is not representing if the word is a verb or not, an adjective or not... It's just the weights that you change to solve your optimization problem to learn the representation of your words.

This tutorial may help : http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/ even though I think the image you put was from this link.

You can also check this, which may help you get started with word vectors with TensorFlow : https://www.tensorflow.org/tutorials/word2vec

| improve this answer | |
$\endgroup$
5
$\begingroup$

TL;DR :

The first matrix represents the input vector in one hot format

The second matrix represents the synaptic weights from the input layer neurons to the hidden layer neurons

Longer Version:

"what the feature matrix exactly is"

It seems you have not understood the representation correctly. That matrix is not a feature matrix but a weight matrix for the neural network. Consider the image given below. Especially notice the left top corner where the Input Layer matrix is multiplied with the Weight matrix. enter image description here

Now look at the top right. This matrix multiplication InputLayer dot-producted with Weights Transpose is just a handy way to represent the neural network at the top right.

So, to answer your question, the equation you have posted is just the mathematical representation for the neural network which is used in the Word2Vec algorithm.

The first part, [0 0 0 1 0 ... 0] represents the input word as a one hot vector and the other matrix represents the weight for the connection of each of the input layer neurons to the hidden layer neurons.

As Word2Vec trains, it backpropogates into these weights and changes them to give better representations of words as vectors.

Once training is complete, you use only this weight matrix, take [0 0 1 0 0 ... 0] for say 'dog' and multiply it with the improved weight matrix to get the vector representation of 'dog' in a dimension = no of hidden layer neurons.

In the diagram you have presented, the number of hidden layer neurons is 3

So the right hand side is basically the word vector.

Image Credits : http://www.datasciencecentral.com/profiles/blogs/matrix-multiplication-in-neural-networks

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.