3
$\begingroup$

So I am new to Deep Learning and NLP. I have read several blog posts on medium, towardsdatascience and papers where they talk about pre-training the word embeddings in an unsupervised fashion and then use them in supervised DNN. But recently I read a blog post which suggested that training the word embeddings while training the neural network gives better results. This is the other link.

So my question is which one should I follow?

Some YouTube videos that I referred:

  1. Deep Learning for NLP without Magic Part 1, 2 and 3
$\endgroup$
3
  • $\begingroup$ which framework are you using? Tensorflow? pytorch? $\endgroup$
    – Escachator
    Jan 28, 2019 at 14:22
  • $\begingroup$ I will be using Tensorflow $\endgroup$ Jan 29, 2019 at 6:18
  • $\begingroup$ Did my answer help? $\endgroup$
    – Escachator
    Jan 30, 2019 at 21:59

2 Answers 2

0
$\begingroup$

There is one answer to know: try both methods and take the one that gives the best result. I would say that in general pre-trained embeddings usually gives better results. You can also start with pre-treained embeddings as initial conditions and let the embeddings train maybe with a smaller learning rate.

In any case, the current state of the art for text classification is ULMFIT (https://arxiv.org/abs/1801.06146), which actually doesn't do any of this. It pre-trains embedding and RNN with a language model in the wikipedia and in the target text and then fine tunes the whole model with the target text.

$\endgroup$
0
$\begingroup$

It depends on access to training data, computational budget, and desired performance level.

Training embedding from scratch will require both training data and computational resources. If you have access to both, there is an increased chance of improved performance for subsequent supervised learning models.

Using pre-trained embeddings require no training, thus no training data or computational resources. Those embedding may or may not be useful for the specific supervised learning task.

One option is a hybrid approach of taking pre-trained embeddings and then fine-tuning them with project-specific data. This has all the advantages of using pre-trained embedding and can leverage task-specific data. This assumes you have access to the necessary computational budget and technical skills.

$\endgroup$

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .