To the best of my understanding , word2vec crated using gensim is of 3 layers only. I was wondering can we customize word2vec NN and create word2vec NN of more than 3 layers to experiment with it using tensorflow ?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Technically, any Word2vec is based on an encoder-decoder Neural Network architecture with a hidden layer that learns the word embedding. In theory it is perfectly feasible to implement a "deeper" word2vec model in TensorFlow.
You are going to find some practical problems, though. Word vectors are super long (for a small corpus, we are in the order to thousands). Training a deep Network with, say, 10.000 input nodes is going to be practically infeasible without an appropriate infrastructure.
To overcome this problem, word2vec libraries (such as gensim) recur to negative sampling. The inventors of word2vec (Mikolov et al.) found that the results from training a simple word2vec model can be approximated by a sequence of logistic regressions. You can find more on negative sampling here and on Google.
But if you make the Network deeper, logistic models (and therefore negative sampling) couldn't work anymore, making this task too computationally intensive for normal computers.
In other words: yeah you can create a word2vec Neural Network with more than 3 layers, but the computational power required makes it almost impossible for individual researchers with common hardware.
