Let’s use a pre-trained model rather than training our own word embeddings. For this, you can download pre-trained vectors from here. Each line of this file contains a word and it’s a corresponding n-dimensional vector. We will create a dictionary using this file for mapping each word to its vector representation.
from gensim.models import FastText
def load_fasttext(): print('loading word embeddings...') embeddings_index = {} f = open('../input/fasttext/wiki.simple.vec',encoding='utf-8') for line in tqdm(f): values = line.strip().rsplit(' ') word = values[0] coefs = np.asarray(values[1:], dtype='float32') embeddings_index[word] = coefs f.close() print('found %s word vectors' % len(embeddings_index))
return embeddings_index
embeddings_index=load_fastext()
Let’s check the embedding for a word,
embeddings_index['london'].shape
Here’s a bit more info on FastText and other document classification methods (for smaller datasets)