2
$\begingroup$

I have a set of reviews from apparel domain, about 100K reviews (2M words). And I want to train word2vec to do some cool NLP staff with it.

However the size is not enough for creating adequate word2vec model, it requires billions of words.

So the idea is to use public corpora (e.g. wikipedia), or even use some pretrained models (e.g. from gensim cool framework) and add my domain specific text. I assume the model will get aware of unseen-in-public words, and could correct vectors for common word.

Does it make sense ? will the 2M words makes any effect at all ?

$\endgroup$
  • $\begingroup$ Yes, you can. Welcome to the site. $\endgroup$ – Emre Aug 21 '18 at 17:07
  • $\begingroup$ Welcome to the site. The 100k reviews is not a small data. word2vec does not require billions of words to work well. You can start training on your own dataset first and check how it performs. $\endgroup$ – hssay Aug 23 '18 at 1:52
2
$\begingroup$

fastText pretrained models should give you a boost to classification task.

gensim on the other hand has possibility to load model and train it with new texts but if you need to account for new words, you need to use

build_vocab(update=True...)

So you can take with fastText pretrained embeddings to gensim and update with your texts.

$\endgroup$
1
$\begingroup$

Yes, you can fine-tune your embeddings while using pre-trained word vectors. If you are using tensorflow, in your tf.get_variable parameter, set trainable=True. The reason it will work is that the only 2M words will be taken from the pre-trained embeddings. Those embeddings will already be close to a local optimum and can be further optimized in very few iterations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.