How to retrain Glove Vectors on top of my own data?

Question

I am using GloVe and gensim for my project. I have a corpus of data (let's say mydata.txt) which has new words which are not in the existing Glove. So, how do I retrain glove so that the existing pre-trained glove must now include the new words on my corpus mydata.txt? I have been struggling and failed to find the solution for 2 weeks. The only similar post I found is this Improving existing GloVe Model

Answer 1

What you should do is:

• Create a new instance of a GloVe model with the old_words and new_words as vocabulary.
• Replace the initial vectors/biases of the old_words with the ones you have already.
• Train this model on mydata.txt.

The new old_words representations won't be the same but will be highly influenced by the old ones.

Answer 2

I am not sure if this is the best method, but this is how I did it. You cannot call it retraining, but this is one of the way you can add on your data with the glove vectors.

from gensim.models import Word2Vec

glove_6b = "glove.6B.100d.txt"

#loading the glove vectors 

with open(glove_6b, "rb") as lines:
     wvec = {
        line.split()[0].decode(encoding): np.array(line.split()[1:], 
                                                         dtype=np.float32)
                                                         for line in lines}

#my data vectors

em_model = Word2Vec(text_data2, size=100, window=5, min_count=1, workers=2)
w2v = {w: vec for w, vec in zip(em_model.wv.index2word, em_model.wv.vectors)}

a = list(w2v.keys())

#mixing them both 
for i in a:
    if i in wvec:
       continue
    else:
       wvec.update({ i  : w2v[i]})

len(list(wvec.keys()))

surely, can be improved. hope this helps.

Answer 3

I don't think the link in the question, nor the idea of adding vectors together are viable.

I believe GloVe (Global Vectors) is not meant to be appended, since it is based on the corpus' overall word co-occurrence statistics from a single corpus known only at initial training time

What you can do is use gensim.scripts.glove2word2vec api to convert GloVe vectors into word2vec, but I don't think you can continue training since its loading in a KeyedVector not a Full Model

Answer 4

If you would like to re-train your specific dataset with a pre-trained glove model you could try this code:

from gensim.test.utils import datapath, get_tmpfile
from gensim.models import KeyedVectors
from gensim.scripts.glove2word2vec import glove2word2vec
import multiprocessing
from tqdm.auto import tqdm

# tokenize your sentences
sentences = [ word_tokenize(sentence) for sentence in tqdm(df.cleaned_text) ]

namaFileGlove = "model.50.glove.txt" #pre train
glove_file = datapath(namaFileGlove)
tmp_file = get_tmpfile("model-wv.50.glove.txt") #the output
glove2word2vec(glove_file, tmp_file)

model_glove = KeyedVectors.load_word2vec_format(tmp_file)

base_model = Word2Vec(vector_size=50, window=5, min_count=5, workers=multiprocessing.cpu_count())
base_model.build_vocab(sentences)
total_examples = base_model.corpus_count

glove_keys = list(model_glove.key_to_index.keys())
base_model.build_vocab([glove_keys], update=True)

base_model.train(sentences, total_examples=total_examples, epochs=base_model.epochs)

base_model.save("model-pretraineed.50.glove")