Looking into word2vec like embeddings I found this exercise on PyTorch's website which prompts the reader to implement a CBOW network in PyTorch.
My question is about the architecture to implement this CBOW network.
Here is my understanding: From a number of sources, it seems that the network should have a single hidden layer (with weights and no biases) which is connected to an activation layer (most sources say softmax). Then the network will be trained to map one-hot encoded words to likely contexts. Finally, the hidden layer's weights will be used to as the embedding matrix.
My confusion is: I see a number of solutions like this first one off google where there are multiple hidden layers. In this example, there is a embedding layer and there are two linear layers connected by a relu. Here is another that uses one linear layer.
My questions are:
- What is the proper architecture to train CBOW encodings?
- If this multiple hidden layer approach is correct, how do you not lose semantic information when you only use one of the layers as the encodings?
- If the single hidden layer approach is correct, does anyone have examples of this being implemented using this approach in PyTorch (fine if no)?
Note: Very new to ML so feel free to correct me even on nit picky things so I can learn!