Language model(LM) is the task of predicting the next word.
Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder.
Or both with-encoder and without-encoder can do the LM task?
$\begingroup$
$\endgroup$
Add a comment
|
$\begingroup$
$\endgroup$
The goal of LM is to learn a probability distribution over sequences of symbols pertaining to a language.
That is, to learn $P(w_1,...,w_N)$ (resource).
This modeling can be accomplished by
- Predicting the next word given the previous words: $P(w_i | w_1,...,w_{i-1})$, or
- Predicting the neighbor words given the center word (Skip-gram): $P(w_{i+k}| w_i), k \in \{-2, -1, 1, 2\}$, or
- Predicting the center word given the neighbor words (CBOW or Continuous Bag-of-Words): $P(w_i| w_{i-2},w_{i-1},w_{i+1}, w_{i+2})$, or other designs.
Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder.
Yes. Modern LM solutions (all deep ones) try to find an encoding (embedding) that helps them to predict the next, neighbor, or center words as close as possible. However, a word encoding can be used as a constant input to other models. The ptb.py code calls text_encoder.TokenTextEncoder to receive such word encodings.
Both with-encoder and without-encoder can do the LM task?
LM task can be tackled without encoders too. For example, we can use frequency tables of adjacent words to build a model (n-gram modeling); e.g. all pairs (We, ?) appeared 10K times, pair (We, can) appeared 100 times, so P(can | We) = 0.01. However, encoder is the core of modern LM solutions.
