# How to initialize a new word2vec model with pre-trained model weights?

I am using Gensim Library in python for using and training word2vector model. Recently, I was looking at initializing my model weights with some pre-trained word2vec model such as (GoogleNewDataset pretrained model). I have been struggling with it couple of weeks. Now, I just searched out that in gesim there is a function that can help me to initialize the weights of my model with pre-trained model weights. That is mentioned below:

reset_from(other_model)

Borrow shareable pre-built structures (like vocab) from the other_model. Useful if testing multiple models in parallel on the same corpus.


• Is the vocabulary of the models same? – Hima Varsha Jul 12 '16 at 7:46
• Why not initiate each of the word2vec parameters with random generated numbers for each run? I could do this and with careful selection of the random numbers for each parameter (numFeatures, contextWindow, seed) I was able to get random similarity tuples which I wanted for my usecase. Simulating an ensemble architecture. What do others think of it? Pls do reply. – zorze Aug 21 '18 at 4:24
• Thanks for your help. It helps me a lot – frhyme Mar 10 at 11:31

Thank Abhishek. I've figure it out! Here are my experiments.

1). we plot a easy example:

from gensim.models import Word2Vec
from sklearn.decomposition import PCA
from matplotlib import pyplot
# define training data
sentences = [['this', 'is', 'the', 'first', 'sentence', 'for', 'word2vec'],
['this', 'is', 'the', 'second', 'sentence'],
['yet', 'another', 'sentence'],
['one', 'more', 'sentence'],
['and', 'the', 'final', 'sentence']]
# train model
model_1 = Word2Vec(sentences, size=300, min_count=1)

# fit a 2d PCA model to the vectors
X = model_1[model_1.wv.vocab]
pca = PCA(n_components=2)
result = pca.fit_transform(X)
# create a scatter plot of the projection
pyplot.scatter(result[:, 0], result[:, 1])
words = list(model_1.wv.vocab)
for i, word in enumerate(words):
pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))
pyplot.show()


From the above plots, we can see that easy sentences cannot distinguish different words' meaning by distances.

from gensim.models import KeyedVectors

model_2 = Word2Vec(size=300, min_count=1)
model_2.build_vocab(sentences)
total_examples = model_2.corpus_count
model_2.build_vocab([list(model.vocab.keys())], update=True)
model_2.intersect_word2vec_format("glove.6B.300d.txt", binary=False, lockf=1.0)
model_2.train(sentences, total_examples=total_examples, epochs=model_2.iter)

# fit a 2d PCA model to the vectors
X = model_2[model_1.wv.vocab]
pca = PCA(n_components=2)
result = pca.fit_transform(X)
# create a scatter plot of the projection
pyplot.scatter(result[:, 0], result[:, 1])
words = list(model_1.wv.vocab)
for i, word in enumerate(words):
pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))
pyplot.show()


From the above figure, we can see that word embeddings are more meaningful.

• this answer is quite informative and helpful in embedding model into a vec file. – Akash Kandpal Jul 13 '18 at 11:38
• @harrypotter0 Thx! – Shixiang Wan Aug 9 '18 at 5:06
• neat and clear mate!!! – vijay athithya Jan 2 '19 at 10:56
• When I attempted to use this, I tested it out with two identical datasets. The outcomes were different for each model. I was hoping that since I would be starting with the same initialized weights, the models would be the same afterwards. How come this was not the case? – Eric Wiener Apr 8 '19 at 1:20
• @EricWiener Because even if training datasets are same, the word vectors for each training are random. The word vector spaces calculated by the same dataset should be similar, and the performance used in NLP tasks should also be similar. – Shixiang Wan Apr 24 '19 at 7:36

Let us look at a sample code:

>>>from gensim.models import word2vec

#let us train a sample model like yours
>>>sentences = [['first', 'sentence'], ['second', 'sentence']]
>>>model1 = word2vec.Word2Vec(sentences, min_count=1)

#let this be the model from which you want to reset
>>>sentences = [['third', 'sentence'], ['fourth', 'sentence']]
>>>model2 = word2vec.Word2Vec(sentences, min_count=1)
>>>model1.reset_from(model2)
>>>model1.similarity('third','sentence')
-0.064622000988260417


Hence, we observe that model1 is being reset by the model2 and hence the word, 'third' and 'sentence' are in it's vocabulary eventually giving its similarity. This is the basic use, you can also check reset_weights() to reset the weights to untrained/initial state.

If you are looking for a pre-trained net for word-embeddings, I would suggest GloVe. The following blog from Keras is very informative of how to implement this. It also has a link to the pre-trained GloVe embeddings. There are pre-trained word vectors ranging from a 50 dimensional vector to 300 dimensional vectors. They were built on either Wikipedia, Common Crawl Data, or Twitter data. You can download them here. Additionally, you should examine the keras blog on how to implement them.

I have done it here in my github repository.

See if this is what you need