can I use public pretrained word2vec, and continue train it for domain specific text?

Question

I have a set of reviews from apparel domain, about 100K reviews (2M words). And I want to train word2vec to do some cool NLP staff with it.

However the size is not enough for creating adequate word2vec model, it requires billions of words.

So the idea is to use public corpora (e.g. wikipedia), or even use some pretrained models (e.g. from gensim cool framework) and add my domain specific text. I assume the model will get aware of unseen-in-public words, and could correct vectors for common word.

Does it make sense ? will the 2M words makes any effect at all ?

Answer 1

fastText pretrained models should give you a boost to classification task.

gensim on the other hand has possibility to load model and train it with new texts but if you need to account for new words, you need to use

build_vocab(update=True...)

So you can take with fastText pretrained embeddings to gensim and update with your texts.

Answer 2

Yes, you can fine-tune your embeddings while using pre-trained word vectors. If you are using tensorflow, in your tf.get_variable parameter, set trainable=True. The reason it will work is that the only 2M words will be taken from the pre-trained embeddings. Those embeddings will already be close to a local optimum and can be further optimized in very few iterations.