I am looking to find a pre-trained weights of an already trained models like Google News data etc. I found it hard to train a new model with enough amount (10 GB etc) of data for myself. So, I want to take benefit from transfer learning in which I would able to get pre-trained layer weights and retrain those weights on my domain specific words. So, definitely it will take relatively less time in training. Any sort of help will be highly appreciated. Thanks in advance :)
Yes, you can take benefit of pre-trained models. Most famous one being the GoogleNewsData trained model which you can find here.
Pre-trained word and phrase vectors https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing
You can then load the vectors in binary format in your model using gensim as shown below.
>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.txt', binary=False) # C text format >>> model = Word2Vec.load_word2vec_format('/tmp/vectors.bin', binary=True) # C binary format
Here is a different pre-built model for English Wikipedia:
Using a prebuilt model
Get python 2.7 Install gensim: pip install gensim uncompress downloaded model: tar -xvf model.tar.gz Load model in gensim: from gensim.models import Word2Vec model = Word2Vec.load("path/to/word2vec/en.model") model.similarity('woman', 'man')
You can also use Stanford NLP Glove
Here is a great compilation of pre-trained word2vec models.
Some additional pre-trained models:
More on gensim and code here: https://radimrehurek.com/gensim/models/word2vec.html
Quora forum with a similar questions
Distributed representations (Glove) based on training on a large corpus are directly available from Stanford NLP group. You can use those word embeddings directly in your application (instead of using 1 hot encoded vectors and then training the network to get the embeddings). If your task is not too specialized starting with this set of embeddings will work well in practice.
It will save you from training an additional $m \times V$ number of parameters where $V$ is the vocabulary size and $m$ is the dimension of the embedding space you want to project into.
Take a look at this paper [PDF]. The main focus is about NER task but the idea is the same - take pre-trained word2vec vectors and adapt them for a specific application.
Many common neural network based applications for NLP frequently start with pre-trained vectors. For example a very recent paper [PDF] (NER and POS tagging tasks) does just this.
from gensim.models import Word2Vec # Word2Vec is full model which is trainable but takes larger memory from gensim.models import KeyedVectors # KeyedVectors is reduced vector model which is NOT trainable but takes less memory model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True) #load pretrained google w2v sen1 = 'w1 w2 w3' sen2 = 'word1 word2 word3' sentences = [[word for word in sen1.split()],[word for word in sen2.split()]] total_examples = model_2.corpus_count model_2 = Word2Vec(size=300, min_count=1) #initiate a full model model_2.build_vocab(sentences) #add words in training dataset #load words from pretrained google dataset model_2.build_vocab([list(model.vocab.keys())], update=True) model_2.intersect_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True, lockf=1.0) #retrain pretrained w2v from new dataset model_2.train(sentences, total_examples=total_examples, epochs=model_2.iter)