What is the structure and dimension of input passed to neural network when training CBOW and SKIP GRAM word embedding

Question

I am confused about input passed to neural network in natural language processing (NLP) when training CBOW word embedding from scratch. I read the paper and have some doubts.

In general neural network (NN) architecture, it is more clear that each row act's as input to neural network with d features. For example in the figure below:

X1, X2, X3 is one input, or one row of the data-frame. So here, one data point is of dimension 3 and data-frame would be like this:

X1  X2  X3
1   2   3
4   5   6
7   8   9

Is my understanding correct?

Now coming to NLP, CBOW architecture: Lets take an example to train CBOW word embeddings:

Sentence1: "I like natural processing domain."

Creating training data from above sentence, window size=1

Input                      output

(I,natural)                like
(like,processing)          natural
(natural,domain)           processing
(processing)               domain

Is the above creation of training data for CBOW architecture for window size=1 correct?

My Questions are below:

How will I pass this training data to neural network for the above figure?

If I represent every word as one-hot encoded vector of dimension equal to size of vocabulary V as input to neural network, then how should I pass 2 words at the same time of dimesion 2V as input.

Is this the way to pass the input for first training sample: I just concatanated the two input words:

Then I train the network to learn word-embeddings using cross entropy loss?

Is this the right way to pass input?

Secondly, the middle layer will give us the word embeddings for 2 input words or the target words??

Answer 1

Just think of Skip-Gram (with negative sampling) as a simple binary logistic classifier.

The data is a collection of nearby word pairs $(w,c)$ extracted from a large corpus. For each of those $k$ negative samples are formed by drawing a context word $c'$ from a noise distribution.

The model has two layers of parameters without a non-linear function between them (equivalent to matrix multiplication of layer parameters) and a sigmoid function on the output (not softmax). Input and output layers have one node per word and the middle layer has the dimension of embeddings $d$ (e.g. 500).

For each word pair $(w, c)$, feed a one-hot vector representing $w$ and predict 1 at the output node representing $c$ and 0 at each of the negative output nodes $c'$.

Each input word $w$ corresponds to $d$ parameters in the first layer and each context word $c$ has $d$ parameters in the second layer. These are embeddings. Since each word is considered as $w$ or $c$ at different times, each dictionary word has two $d$-dimensional embeddings (one in each layer), typically only one is used.