Language model(LM) is the task of predicting the next word.
Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder.
Or both with-encoder and without-encoder can do the LM task?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this communityLanguage model(LM) is the task of predicting the next word.
Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder.
Or both with-encoder and without-encoder can do the LM task?
The goal of LM is to learn a probability distribution over sequences of symbols pertaining to a language.
That is, to learn $P(w_1,...,w_N)$ (resource).
This modeling can be accomplished by
Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder.
Yes. Modern LM solutions (all deep ones) try to find an encoding (embedding) that helps them to predict the next, neighbor, or center words as close as possible. However, a word encoding can be used as a constant input to other models. The ptb.py code calls text_encoder.TokenTextEncoder
to receive such word encodings.
Both with-encoder and without-encoder can do the LM task?
LM task can be tackled without encoders too. For example, we can use frequency tables of adjacent words to build a model (n-gram modeling); e.g. all pairs (We, ?) appeared 10K times, pair (We, can) appeared 100 times, so P(can | We) = 0.01. However, encoder is the core of modern LM solutions.