# Can we take of benefit of using transfer learning while training a word2vec models?

I am looking to find a pre-trained weights of an already trained models like Google News data etc. I found it hard to train a new model with enough amount (10 GB etc) of data for myself. So, I want to take benefit from transfer learning in which I would able to get pre-trained layer weights and retrain those weights on my domain specific words. So, definitely it will take relatively less time in training. Any sort of help will be highly appreciated. Thanks in advance :)

Yes, you can take benefit of pre-trained models. Most famous one being the GoogleNewsData trained model which you can find here.

Pre-trained word and phrase vectors https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing

You can then load the vectors in binary format in your model using gensim as shown below.

>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.txt', binary=False)  # C text format
>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.bin', binary=True)  # C binary format


Here is a different pre-built model for English Wikipedia:

https://github.com/idio/wiki2vec/raw/master/torrents/enwiki-gensim-word2vec-1000-nostem-10cbow.torrent

Using a prebuilt model

Get python 2.7
Install gensim: pip install gensim
from gensim.models import Word2Vec
model.similarity('woman', 'man')


You can also use Stanford NLP Glove

Here is a great compilation of pre-trained word2vec models.

More on gensim and code here: https://radimrehurek.com/gensim/models/word2vec.html

Quora forum with a similar questions

• Okay, that is great piece of information good work. But, can i use pre-trained models layer weights to initialize a new model and then further tune that model with my sentences? – Nomiluks Mar 15 '16 at 18:04
• @Nomi Yes. From [gensim documentation] (radimrehurek.com/gensim/models/word2vec.html) , once you load the model, model = Word2Vec.load(fname) # you can continue training with the loaded model! – Guru Mar 20 '16 at 14:25
• Documentation link above mentions: "NOTE: It is impossible to continue training the vectors loaded from the C format because hidden weights, vocabulary frequency and the binary tree is missing." – trailblazer Mar 27 '17 at 2:27

Distributed representations (Glove) based on training on a large corpus are directly available from Stanford NLP group. You can use those word embeddings directly in your application (instead of using 1 hot encoded vectors and then training the network to get the embeddings). If your task is not too specialized starting with this set of embeddings will work well in practice.

It will save you from training an additional $m \times V$ number of parameters where $V$ is the vocabulary size and $m$ is the dimension of the embedding space you want to project into.

• But, i want to initialize my new word2vec model with pre-trained model weights. Is it possible to use already pre-trained model layer weights to initialize new model. After initialization i want to train that model with new sentences. is it possible? – Nomiluks Mar 15 '16 at 18:07
• Yes you can. However I don't think the weight matrix is available publicly – wabbit Mar 19 '16 at 3:38
• Yup, right...? If we train a model ourselves and try to access the access the trained model weights using Gensim library. Is it possible – Nomiluks Mar 19 '16 at 7:49
• Not sure about gensim but because it's a parameter to be optimized most software should allow it – wabbit Mar 19 '16 at 11:04
• @HrishikeshGanu Is this link still working? github.com/idio/wiki2vec/raw/master/torrents/… Source: github.com/idio/wiki2vec – Anish May 5 '17 at 20:40

Take a look at this paper [PDF]. The main focus is about NER task but the idea is the same - take pre-trained word2vec vectors and adapt them for a specific application.

Many common neural network based applications for NLP frequently start with pre-trained vectors. For example a very recent paper [PDF] (NER and POS tagging tasks) does just this.

from gensim.models import Word2Vec
# Word2Vec is full model which is trainable but takes larger memory

from gensim.models import KeyedVectors
# KeyedVectors is reduced vector model which is NOT trainable but takes less memory

sen1 = 'w1 w2 w3'
sen2 = 'word1 word2 word3'
sentences = [[word for word in sen1.split()],[word for word in sen2.split()]]
total_examples = model_2.corpus_count

model_2 = Word2Vec(size=300, min_count=1) #initiate a full model
model_2.build_vocab(sentences) #add words in training dataset