Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I'm a beginner in neural networks and currently I'm exploring the word2vec model. However I'm having a tough time to understand what the feature matrix exactly is.
I can understand that the first matrix is a one-hot encoding vector for a given word, but what does the second matrix signify? More specifically, what does each of those values (ie. 17, 24, 1 etc) mean?
The idea behind word2vec is to represent words by a vector of real numbers of dimension d. Therefore the second matrix is the representation of those words.
The i-th line of this matrix is the vector representation of the i-th word.
Let's say that in your example you have 5 words : ["Lion", "Cat", "Dog", "Horse", "Mouse"], then the first vector [0,0,0,1,0] means you're considering the word "Horse" and so the representation of "Horse" is [10, 12, 19]. Similarly, [17, 24, 1] is the representation of the word "Lion".
To my knowledege, there are no "human meaning" specifically to each of the numbers in these representations. One number is not representing if the word is a verb or not, an adjective or not... It's just the weights that you change to solve your optimization problem to learn the representation of your words.
This tutorial may help : http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/ even though I think the image you put was from this link.
You can also check this, which may help you get started with word vectors with TensorFlow : https://www.tensorflow.org/tutorials/word2vec
TL;DR :
The first matrix represents the input vector in one hot format
The second matrix represents the synaptic weights from the input layer neurons to the hidden layer neurons
Longer Version:
"what the feature matrix exactly is"
It seems you have not understood the representation correctly.
That matrix is not a feature matrix but a weight matrix for the neural network.
Consider the image given below. Especially notice the left top corner where the Input Layer matrix is multiplied with the Weight matrix.
Now look at the top right. This matrix multiplication InputLayer dot-producted with Weights Transpose is just a handy way to represent the neural network at the top right.
So, to answer your question, the equation you have posted is just the mathematical representation for the neural network which is used in the Word2Vec algorithm.
The first part, [0 0 0 1 0 ... 0] represents the input word as a one hot vector and the other matrix represents the weight for the connection of each of the input layer neurons to the hidden layer neurons.
As Word2Vec trains, it backpropogates into these weights and changes them to give better representations of words as vectors.
Once training is complete, you use only this weight matrix, take [0 0 1 0 0 ... 0] for say 'dog' and multiply it with the improved weight matrix to get the vector representation of 'dog' in a dimension = no of hidden layer neurons.
In the diagram you have presented, the number of hidden layer neurons is 3
So the right hand side is basically the word vector.
Image Credits : http://www.datasciencecentral.com/profiles/blogs/matrix-multiplication-in-neural-networks
