Continuous Bag Of Words (CBOW) network architecture?

Question

Looking into word2vec like embeddings I found this exercise on PyTorch's website which prompts the reader to implement a CBOW network in PyTorch.

My question is about the architecture to implement this CBOW network.

Here is my understanding: From a number of sources, it seems that the network should have a single hidden layer (with weights and no biases) which is connected to an activation layer (most sources say softmax). Then the network will be trained to map one-hot encoded words to likely contexts. Finally, the hidden layer's weights will be used to as the embedding matrix.

My confusion is: I see a number of solutions like this first one off google where there are multiple hidden layers. In this example, there is a embedding layer and there are two linear layers connected by a relu. Here is another that uses one linear layer.

My questions are:

• What is the proper architecture to train CBOW encodings?
• If this multiple hidden layer approach is correct, how do you not lose semantic information when you only use one of the layers as the encodings?
• If the single hidden layer approach is correct, does anyone have examples of this being implemented using this approach in PyTorch (fine if no)?

Note: Very new to ML so feel free to correct me even on nit picky things so I can learn!

Answer 1

• What is the proper architecture to train CBOW encodings?

The original paper by Mikolov et al uses 1 hidden layer. However, for NLP tasks (and deep learning in general), there is no "correct" number of layers to use.

For some tasks, you might find using more hidden layers to be better, for other tasks maybe one hidden layer is sufficient. The number of layers is a hyperparameter, along with other 'design choices' like the dimension of the embedding vectors, the size of the vocabulary, and many, many more.

• If this multiple hidden layer approach is correct, how do you not lose semantic information when you only use one of the layers as the encodings?

I think of it this way. When training CBOW, the hidden layers learn some 'relationship function' between the input context words and the output target word. In order for the 'relationship function' to perform well, the embeddings must be arranged accordingly (in N-dimensional space), and it just happens that the optimal arrangement encodes semantic information.

So the hidden layers shouldn't really have anything to do with semantic information. However, because of the immense modeling power of hidden layers (which make up neural networks), the truth is the embeddings will inevitably lose some semantic information to the hidden layers.

• If the single hidden layer approach is correct, does anyone have examples of this being implemented using this approach in PyTorch (fine if no)?

Here's literally the 1st github example off Google.