0
$\begingroup$

I'm trying to understand the relationship between pretrained word vectors and pretrained weights when using pretrained word vectors in another neural network.

Are the vectors themselves the weights or do the weights need to be transferred in addition to selecting embedded representations? What's the high level conceptual process in using pretrained vectors?

Thanks!

$\endgroup$

1 Answer 1

1
$\begingroup$

Pre-trained word embedding methods such as word2vec, are used in NLP for the initialization of the first layer of a neural network before training for a new task. It is a single layer of weights (known as embeddings) and any prior knowledge from the word embeddings is only present in the first layer of the network, so the entire network would still need to be trained for a new target task.

$\endgroup$
2
  • $\begingroup$ Thanks. can they also be used as a feature vector? $\endgroup$
    – taiidan
    Jan 16, 2019 at 21:16
  • $\begingroup$ It depends on what you want to do. Pretrained embeddings come from an unsupervised procedure and if you want to solve e.g. a classification problem you have to come up with a supervised learning solution. So you you have to train you model from scratch using the pretrained embeddings for the initialization of the first layer. $\endgroup$ Jan 16, 2019 at 23:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.