1
$\begingroup$

When using pretrained GloVe.6B for embedding generation, How can I get only the top most frequently used 100000 words rather than all the 4M words in the file?

$\endgroup$

2 Answers 2

2
$\begingroup$

I was stuck in a similar problem while working with glove. Assuming that you have a dataset in text form, from which you want to collect the topmost 100000 words, you'll have to make a list of those words. In the glove file, each embedding is on a separate line, with each line starting with the word itself and then the embedding. You'll have to write a code to compare your list of words with the words in glove file and extract the lines which make a hit. Have a look here for example code.

$\endgroup$
2
  • $\begingroup$ This is very obvious, but will increase the time consumption of the algorithm MxN folds. $\endgroup$
    – thanatoz
    Commented Oct 1, 2018 at 10:47
  • $\begingroup$ Yes, it will take a lot of time, but since this only needs to be run once I used this. $\endgroup$
    – bkshi
    Commented Oct 1, 2018 at 10:53
0
$\begingroup$

You can try this method:

from keras.preprocessing.text import Tokenizer
from gensim.models import KeyedVectors

# X is the corpus
# GLOVE_DIR is the glove model
# EMBEDDING_DIM is the embedding demension of glove model

VOVAB_SIZE = 10000
tokenizer = Tokenizer()
tokenizer.fit_on_texts(X)
word_index = tokenizer.word_index

glove_model = KeyedVectors.load_word2vec_format(GLOVE_DIR, binary=True)

num_words = min(VOCAB_SIZE, len(word_index) + 1)
embedding_matrix = np.zeros((len(num_words) + 1, EMBEDDING_DIM))

for word, i in word_index.items():
    if i < VOVAB_SIZE:
        if word in set(glove_model.wv.index2word):
            embedding_matrix[i] = glove_model[word]
        else:
            embedding_matrix[i] = np.random.rand(1, EMBEDDING_DIM)

the embedding_matrix is the most frequent 10000 words in your corpus

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.