Might not be the answer you are seeking, but I'll still have a go:
First, quick review of word2Vec, assume we are using skip gram.
A typical Word2Vec train-able model consists of 1 input layer (for example, 10 000 long one-hot vector), a hidden layer (for example 300 neurons), an output (10 000 long one-hot vector)
- Input: 10 000
- Hidden: 300
- Output: 10 000
There is a matrix E between Input-Hidden, describing the weights to make your one-hot into an embedding. The matrix is special because each column (or rows, depending on your preferred notation) represents pre-activations in those 300 neurons - a response to a corresponding incoming 1-hot vector.
You don't need to perform any activation on these 300 neurons and can use their values straight away as an embedding in any future task.
However, simply squeezing a one-hot into a 300-dimensional representation isn't enough - it must have a meaning. And we ensure this meaning is correct using an additional second matrix - which connects Hidden to Output
We don't want to activate a hidden layer because activation-function won't be needed during runtime, however, in that case we will need a second matrix, going from Hidden to Output.
This second matrix will make an entirely different one-hot from your embeding. Such a one-hot will represent a most likely word to be nearby (contextually) of your original one-hot. In other words, this output won't be your original one-hot.
That's why a second matrix is needed.
At the output, we perform a softmax, like in a classification problem.
This allows us to express a relation "word"-->embedding-->"context-neighbor-word"
Now, backpropagation can be done, to correct the Input-Hidden weights (Your first matrix E) - these are the weights we really care about. That's because Matrix E can be used during Runtime (I think), perhaps being plugged as a first fully-connected layer into some Recurrent Neural Net.
In that case you wouldn't use this:
You don't need to perform any activation on these 300 neurons and can use their values straight away as an embedding in any future task
but instead, you would just grab the appropriate column (or row, depending on your preferred notation) from that matrix, during Runtime. The benefit is that this way you get a very cheaply pre-trained fully-connected layer, designed to work with one-hots. Usually, first layers would be longest to train, due to the vanishing-gradient issue.
Why a second matrix is needed during training:
Once again, recall, there is no activation at the hidden layer.
We can dictate the network what "one-hot" must have been created in response to your original "input one-hot", and can punish network if it fails to generate a correct answer.
We cannot put softmax directly after hidden layer, because we are interested in summoning a mechanism to convert into an embedding. That's already a responsibility of a first matrix E. Thus, we need an extra step (an extra matrix) that will give us enough room to now form a conclusion at the output layer about a different but similar (context-wise) neighbor-word
During the runtime you throw away the second matrix. But don't delete it permanently in case if you need to come back and continue training your model.