I am looking to find a pre-trained weights of an already trained models like Google News data etc. I found it hard to train a new model with enough amount (10 GB etc) of data for myself. So, I want to take benefit from transfer learning in which I would able to get pre-trained layer weights and retrain those weights on my domain specific words. So, definitely it will take relatively less time in training. Any sort of help will be highly appreciated. Thanks in advance :)
$\begingroup$
$\endgroup$
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
3
Yes, you can take benefit of pre-trained models. Most famous one being the GoogleNewsData trained model which you can find here.
Pre-trained word and phrase vectors https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing
You can then load the vectors in binary format in your model using gensim as shown below.
>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.txt', binary=False) # C text format
>>> model = Word2Vec.load_word2vec_format('/tmp/vectors.bin', binary=True) # C binary format
Here is a different pre-built model for English Wikipedia:
Source: https://github.com/idio/wiki2vec/
Using a prebuilt model
Get python 2.7
Install gensim: pip install gensim
uncompress downloaded model: tar -xvf model.tar.gz
Load model in gensim:
from gensim.models import Word2Vec
model = Word2Vec.load("path/to/word2vec/en.model")
model.similarity('woman', 'man')
You can also use Stanford NLP Glove
Here is a great compilation of pre-trained word2vec models.
Some additional pre-trained models:
More on gensim and code here: https://radimrehurek.com/gensim/models/word2vec.html
Quora forum with a similar questions
-
2$\begingroup$ Okay, that is great piece of information good work. But, can i use pre-trained models layer weights to initialize a new model and then further tune that model with my sentences? $\endgroup$– NomiluksMar 15, 2016 at 18:04
-
$\begingroup$ @Nomi Yes. From [gensim documentation] (radimrehurek.com/gensim/models/word2vec.html) , once you load the model,
model = Word2Vec.load(fname) # you can continue training with the loaded model!$\endgroup$– GuruMar 20, 2016 at 14:25 -
3$\begingroup$ Documentation link above mentions: "NOTE: It is impossible to continue training the vectors loaded from the C format because hidden weights, vocabulary frequency and the binary tree is missing." $\endgroup$ Mar 27, 2017 at 2:27
$\begingroup$
$\endgroup$
5
Distributed representations (Glove) based on training on a large corpus are directly available from Stanford NLP group. You can use those word embeddings directly in your application (instead of using 1 hot encoded vectors and then training the network to get the embeddings). If your task is not too specialized starting with this set of embeddings will work well in practice.
It will save you from training an additional $m \times V$ number of parameters where $V$ is the vocabulary size and $m$ is the dimension of the embedding space you want to project into.
-
$\begingroup$ But, i want to initialize my new word2vec model with pre-trained model weights. Is it possible to use already pre-trained model layer weights to initialize new model. After initialization i want to train that model with new sentences. is it possible? $\endgroup$– NomiluksMar 15, 2016 at 18:07
-
$\begingroup$ Yes you can. However I don't think the weight matrix is available publicly $\endgroup$– wabbitMar 19, 2016 at 3:38
-
$\begingroup$ Yup, right...? If we train a model ourselves and try to access the access the trained model weights using Gensim library. Is it possible $\endgroup$– NomiluksMar 19, 2016 at 7:49
-
$\begingroup$ Not sure about gensim but because it's a parameter to be optimized most software should allow it $\endgroup$– wabbitMar 19, 2016 at 11:04
-
$\begingroup$ @HrishikeshGanu Is this link still working? github.com/idio/wiki2vec/raw/master/torrents/… Source: github.com/idio/wiki2vec $\endgroup$– AnishMay 5, 2017 at 20:40
$\begingroup$
$\endgroup$
0
Take a look at this paper [PDF]. The main focus is about NER task but the idea is the same - take pre-trained word2vec vectors and adapt them for a specific application.
Many common neural network based applications for NLP frequently start with pre-trained vectors. For example a very recent paper [PDF] (NER and POS tagging tasks) does just this.
answered Mar 11, 2016 at 1:12
$\begingroup$
$\endgroup$
from gensim.models import Word2Vec
# Word2Vec is full model which is trainable but takes larger memory
from gensim.models import KeyedVectors
# KeyedVectors is reduced vector model which is NOT trainable but takes less memory
model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True) #load pretrained google w2v
sen1 = 'w1 w2 w3'
sen2 = 'word1 word2 word3'
sentences = [[word for word in sen1.split()],[word for word in sen2.split()]]
total_examples = model_2.corpus_count
model_2 = Word2Vec(size=300, min_count=1) #initiate a full model
model_2.build_vocab(sentences) #add words in training dataset
#load words from pretrained google dataset
model_2.build_vocab([list(model.vocab.keys())], update=True)
model_2.intersect_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True, lockf=1.0)
#retrain pretrained w2v from new dataset
model_2.train(sentences, total_examples=total_examples, epochs=model_2.iter)
