2
$\begingroup$

I am using GloVe and gensim for my project. I have a corpus of data (let's say mydata.txt) which has new words which are not in the existing Glove. So, how do I retrain glove so that the existing pre-trained glove must now include the new words on my corpus mydata.txt? I have been struggling and failed to find the solution for 2 weeks. The only similar post I found is this Improving existing GloVe Model

$\endgroup$
2
  • 1
    $\begingroup$ Did you try the suggested solution that you linked to? $\endgroup$
    – Tom M.
    Jun 29, 2018 at 12:46
  • $\begingroup$ Please tell us what code you use, there is no implementation of GloVe in Gensim. $\endgroup$
    – Robin
    Jun 29, 2018 at 13:23

4 Answers 4

1
$\begingroup$

What you should do is:

  • Create a new instance of a GloVe model with the old_words and new_words as vocabulary.
  • Replace the initial vectors/biases of the old_words with the ones you have already.
  • Train this model on mydata.txt.

The new old_words representations won't be the same but will be highly influenced by the old ones.

$\endgroup$
1
$\begingroup$

I am not sure if this is the best method, but this is how I did it. You cannot call it retraining, but this is one of the way you can add on your data with the glove vectors.

from gensim.models import Word2Vec

glove_6b = "glove.6B.100d.txt"

#loading the glove vectors 

with open(glove_6b, "rb") as lines:
     wvec = {
        line.split()[0].decode(encoding): np.array(line.split()[1:], 
                                                         dtype=np.float32)
                                                         for line in lines}

#my data vectors

em_model = Word2Vec(text_data2, size=100, window=5, min_count=1, workers=2)
w2v = {w: vec for w, vec in zip(em_model.wv.index2word, em_model.wv.vectors)}

a = list(w2v.keys())

#mixing them both 
for i in a:
    if i in wvec:
       continue
    else:
       wvec.update({ i  : w2v[i]})

len(list(wvec.keys()))

surely, can be improved. hope this helps.

$\endgroup$
1
  • $\begingroup$ Is this the right approach? Have you got some improvements with this one? $\endgroup$
    – Deshwal
    Dec 23, 2021 at 9:54
1
$\begingroup$

I don't think the link in the question, nor the idea of adding vectors together are viable.

I believe GloVe (Global Vectors) is not meant to be appended, since it is based on the corpus' overall word co-occurrence statistics from a single corpus known only at initial training time

What you can do is use gensim.scripts.glove2word2vec api to convert GloVe vectors into word2vec, but I don't think you can continue training since its loading in a KeyedVector not a Full Model

$\endgroup$
0
$\begingroup$

If you would like to re-train your specific dataset with a pre-trained glove model you could try this code:

from gensim.test.utils import datapath, get_tmpfile
from gensim.models import KeyedVectors
from gensim.scripts.glove2word2vec import glove2word2vec
import multiprocessing
from tqdm.auto import tqdm

# tokenize your sentences
sentences = [ word_tokenize(sentence) for sentence in tqdm(df.cleaned_text) ]

namaFileGlove = "model.50.glove.txt" #pre train
glove_file = datapath(namaFileGlove)
tmp_file = get_tmpfile("model-wv.50.glove.txt") #the output
glove2word2vec(glove_file, tmp_file)

model_glove = KeyedVectors.load_word2vec_format(tmp_file)

base_model = Word2Vec(vector_size=50, window=5, min_count=5, workers=multiprocessing.cpu_count())
base_model.build_vocab(sentences)
total_examples = base_model.corpus_count

glove_keys = list(model_glove.key_to_index.keys())
base_model.build_vocab([glove_keys], update=True)

base_model.train(sentences, total_examples=total_examples, epochs=base_model.epochs)

base_model.save("model-pretraineed.50.glove")
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.