How to obtain the word vectors optimally

Question

I have a list of strings as shown

sent_list = ["Carrefour is in France", "Apple pie is delicious", "Amazon has just delivered", ...]

My code to get word embeddings below

import spacy

nlp = spacy.load("en_trf_bertbaseuncased_lg")

for sent in sent_list:
    print(nlp(sent).vector)

This takes considerable time when the list is of large size (>10000). I tried disabling sentencizer within the nlp pipe but with not much improvement. How can this be optimized for shorter run time?

Answer 1

Use nlp.pipe() to process texts in larger batches, which is much faster, especially for a lot of short texts:

for doc in nlp.pipe(sent_list):
    # averaged doc vector
    print(doc.vector)
    # token vectors
    print([token.vector for token in doc])

Answer 2

Yep, spacy is known to be a bit slow. Check this answer to similar problem. It'll be bit faster, once you run torch model under the hood. Additionally, you don't have to worry about language. You can easily detect it using libs like langdetect. Language codes follow ISO standard, so no worries there too.