3
$\begingroup$

So I am new to Deep Learning and NLP. I have read several blog posts on medium, towardsdatascience and papers where they talk about pre-training the word embeddings in an unsupervised fashion and then use them in supervised DNN. But recently I read a blog post (Link 1) which suggested that training the word embeddings while training the neural network gives better results.

So my question is which one should I follow?

Links that I referred to:

  1. https://machinelearningmastery.com/develop-word-embedding-model-predicting-movie-review-sentiment

  2. https://towardsdatascience.com/deep-learning-for-specific-information-extraction-from-unstructured-texts-12c5b9dceada

Some YouTube videos that I referred:

  1. Deep Learning for NLP without Magic Part 1, 2 and 3
$\endgroup$
  • $\begingroup$ which framework are you using? Tensorflow? pytorch? $\endgroup$ – Escachator Jan 28 at 14:22
  • $\begingroup$ I will be using Tensorflow $\endgroup$ – Vatsal Aggarwal Jan 29 at 6:18
  • $\begingroup$ Did my answer help? $\endgroup$ – Escachator Jan 30 at 21:59
0
$\begingroup$

There is one answer to know: try both methods and take the one that gives the best result. I would say that in general pre-trained embeddings usually gives better results. You can also start with pre-treained embeddings as initial conditions and let the embeddings train maybe with a smaller learning rate.

In any case, the current state of the art for text classification is ULMFIT (https://arxiv.org/abs/1801.06146), which actually doesn't do any of this. It pre-trains embedding and RNN with a language model in the wikipedia and in the target text and then fine tunes the whole model with the target text.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.