15
$\begingroup$

I am using Gensim Library in python for using and training word2vector model. Recently, I was looking at initializing my model weights with some pre-trained word2vec model such as (GoogleNewDataset pretrained model). I have been struggling with it couple of weeks. Now, I just searched out that in gesim there is a function that can help me to initialize the weights of my model with pre-trained model weights. That is mentioned below:

reset_from(other_model)

    Borrow shareable pre-built structures (like vocab) from the other_model. Useful if testing multiple models in parallel on the same corpus.

I don't know this function can do the same thing or not. Please help!!!

$\endgroup$
  • $\begingroup$ Is the vocabulary of the models same? $\endgroup$ – Hima Varsha Jul 12 '16 at 7:46
  • $\begingroup$ Why not initiate each of the word2vec parameters with random generated numbers for each run? I could do this and with careful selection of the random numbers for each parameter (numFeatures, contextWindow, seed) I was able to get random similarity tuples which I wanted for my usecase. Simulating an ensemble architecture. What do others think of it? Pls do reply. $\endgroup$ – zorze Aug 21 '18 at 4:24
  • $\begingroup$ Thanks for your help. It helps me a lot $\endgroup$ – frhyme Mar 10 at 11:31
18
$\begingroup$

Thank Abhishek. I've figure it out! Here are my experiments.

1). we plot a easy example:

from gensim.models import Word2Vec
from sklearn.decomposition import PCA
from matplotlib import pyplot
# define training data
sentences = [['this', 'is', 'the', 'first', 'sentence', 'for', 'word2vec'],
            ['this', 'is', 'the', 'second', 'sentence'],
            ['yet', 'another', 'sentence'],
            ['one', 'more', 'sentence'],
            ['and', 'the', 'final', 'sentence']]
# train model
model_1 = Word2Vec(sentences, size=300, min_count=1)

# fit a 2d PCA model to the vectors
X = model_1[model_1.wv.vocab]
pca = PCA(n_components=2)
result = pca.fit_transform(X)
# create a scatter plot of the projection
pyplot.scatter(result[:, 0], result[:, 1])
words = list(model_1.wv.vocab)
for i, word in enumerate(words):
    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))
pyplot.show()

enter image description here

From the above plots, we can see that easy sentences cannot distinguish different words' meaning by distances.

2). Load pre-trained word embedding:

from gensim.models import KeyedVectors

model_2 = Word2Vec(size=300, min_count=1)
model_2.build_vocab(sentences)
total_examples = model_2.corpus_count
model = KeyedVectors.load_word2vec_format("glove.6B.300d.txt", binary=False)
model_2.build_vocab([list(model.vocab.keys())], update=True)
model_2.intersect_word2vec_format("glove.6B.300d.txt", binary=False, lockf=1.0)
model_2.train(sentences, total_examples=total_examples, epochs=model_2.iter)

# fit a 2d PCA model to the vectors
X = model_2[model_1.wv.vocab]
pca = PCA(n_components=2)
result = pca.fit_transform(X)
# create a scatter plot of the projection
pyplot.scatter(result[:, 0], result[:, 1])
words = list(model_1.wv.vocab)
for i, word in enumerate(words):
    pyplot.annotate(word, xy=(result[i, 0], result[i, 1]))
pyplot.show()

enter image description here

From the above figure, we can see that word embeddings are more meaningful.
Hope this answer will be helpful.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ this answer is quite informative and helpful in embedding model into a vec file. $\endgroup$ – Akash Kandpal Jul 13 '18 at 11:38
  • $\begingroup$ @harrypotter0 Thx! $\endgroup$ – Shixiang Wan Aug 9 '18 at 5:06
  • $\begingroup$ neat and clear mate!!! $\endgroup$ – vijay athithya Jan 2 '19 at 10:56
  • $\begingroup$ When I attempted to use this, I tested it out with two identical datasets. The outcomes were different for each model. I was hoping that since I would be starting with the same initialized weights, the models would be the same afterwards. How come this was not the case? $\endgroup$ – Eric Wiener Apr 8 '19 at 1:20
  • 1
    $\begingroup$ @EricWiener Because even if training datasets are same, the word vectors for each training are random. The word vector spaces calculated by the same dataset should be similar, and the performance used in NLP tasks should also be similar. $\endgroup$ – Shixiang Wan Apr 24 '19 at 7:36
4
$\begingroup$

Let us look at a sample code:

>>>from gensim.models import word2vec

#let us train a sample model like yours
>>>sentences = [['first', 'sentence'], ['second', 'sentence']]
>>>model1 = word2vec.Word2Vec(sentences, min_count=1)

#let this be the model from which you want to reset
>>>sentences = [['third', 'sentence'], ['fourth', 'sentence']]
>>>model2 = word2vec.Word2Vec(sentences, min_count=1)
>>>model1.reset_from(model2)
>>>model1.similarity('third','sentence')
-0.064622000988260417

Hence, we observe that model1 is being reset by the model2 and hence the word, 'third' and 'sentence' are in it's vocabulary eventually giving its similarity. This is the basic use, you can also check reset_weights() to reset the weights to untrained/initial state.

| improve this answer | |
$\endgroup$
2
$\begingroup$

If you are looking for a pre-trained net for word-embeddings, I would suggest GloVe. The following blog from Keras is very informative of how to implement this. It also has a link to the pre-trained GloVe embeddings. There are pre-trained word vectors ranging from a 50 dimensional vector to 300 dimensional vectors. They were built on either Wikipedia, Common Crawl Data, or Twitter data. You can download them here. Additionally, you should examine the keras blog on how to implement them.

| improve this answer | |
$\endgroup$
1
$\begingroup$

I have done it here in my github repository.

See if this is what you need

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.