I am confused about input passed to neural network in natural language processing (NLP) when training CBOW word embedding from scratch. I read the paper and have some doubts.

In general neural network (NN) architecture, it is more clear that each row act's as input to neural network with d features. For example in the figure below:

enter image description here

X1, X2, X3 is one input, or one row of the data-frame. So here, one data point is of dimension 3 and data-frame would be like this:

X1  X2  X3
1   2   3
4   5   6
7   8   9

Is my understanding correct?

Now coming to NLP, CBOW architecture: Lets take an example to train CBOW word embeddings:

Sentence1: "I like natural processing domain."

Creating training data from above sentence, window size=1

Input                      output

(I,natural)                like
(like,processing)          natural
(natural,domain)           processing
(processing)               domain

Is the above creation of training data for CBOW architecture for window size=1 correct?

My Questions are below:

How will I pass this training data to neural network for the above figure?

If I represent every word as one-hot encoded vector of dimension equal to size of vocabulary V as input to neural network, then how should I pass 2 words at the same time of dimesion 2V as input.

Is this the way to pass the input for first training sample: I just concatanated the two input words:

enter image description here

Then I train the network to learn word-embeddings using cross entropy loss?

Is this the right way to pass input?

Secondly, the middle layer will give us the word embeddings for 2 input words or the target words??


1 Answer 1


Just think of Skip-Gram (with negative sampling) as a simple binary logistic classifier.

The data is a collection of nearby word pairs $(w,c)$ extracted from a large corpus. For each of those $k$ negative samples are formed by drawing a context word $c'$ from a noise distribution.

The model has two layers of parameters without a non-linear function between them (equivalent to matrix multiplication of layer parameters) and a sigmoid function on the output (not softmax). Input and output layers have one node per word and the middle layer has the dimension of embeddings $d$ (e.g. 500).

For each word pair $(w, c)$, feed a one-hot vector representing $w$ and predict 1 at the output node representing $c$ and 0 at each of the negative output nodes $c'$.

Each input word $w$ corresponds to $d$ parameters in the first layer and each context word $c$ has $d$ parameters in the second layer. These are embeddings. Since each word is considered as $w$ or $c$ at different times, each dictionary word has two $d$-dimensional embeddings (one in each layer), typically only one is used.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.