Help with reusing glove word embedding pretrained model

Question

When using pretrained GloVe.6B for embedding generation, How can I get only the top most frequently used 100000 words rather than all the 4M words in the file?

Answer 1

I was stuck in a similar problem while working with glove. Assuming that you have a dataset in text form, from which you want to collect the topmost 100000 words, you'll have to make a list of those words. In the glove file, each embedding is on a separate line, with each line starting with the word itself and then the embedding. You'll have to write a code to compare your list of words with the words in glove file and extract the lines which make a hit. Have a look here for example code.

Answer 2

You can try this method:

from keras.preprocessing.text import Tokenizer
from gensim.models import KeyedVectors

# X is the corpus
# GLOVE_DIR is the glove model
# EMBEDDING_DIM is the embedding demension of glove model

VOVAB_SIZE = 10000
tokenizer = Tokenizer()
tokenizer.fit_on_texts(X)
word_index = tokenizer.word_index

glove_model = KeyedVectors.load_word2vec_format(GLOVE_DIR, binary=True)

num_words = min(VOCAB_SIZE, len(word_index) + 1)
embedding_matrix = np.zeros((len(num_words) + 1, EMBEDDING_DIM))

for word, i in word_index.items():
    if i < VOVAB_SIZE:
        if word in set(glove_model.wv.index2word):
            embedding_matrix[i] = glove_model[word]
        else:
            embedding_matrix[i] = np.random.rand(1, EMBEDDING_DIM)

the embedding_matrix is the most frequent 10000 words in your corpus