0
$\begingroup$

I'm new to LeakGAN or SeqGAN or TextGAN. I know GAN is to generate text and let discriminator un-judge-able to real text and gen-text.

LM(language model) is the task of predicting the next word and can also be used to generate text.

$\endgroup$
0
$\begingroup$
  • LM are trained on the ground truth (predict the next symbol from the true past symbols) but deployed in a different setting (when generating a string, the LM predicts the next symbol from the generated past symbols) such difference between training and deployment does no occur with GAN

  • The main idea of ​​the use of the two methods may be one, but the working principle is different. Note that LM may follow the statistical fundamentals to reach the result. Either the GAN method depends on the networks.

The work of these two methods may also differ as data types work.

  • LM based on HMM that take account into the statistical relationship of words, while GAN works as the black-box method using abstract features (not human-reachable)

I believe that is the correct definition of a language model, whether the predicted next token is a character or a word. There are many statistical language models that do rely on probability distributions over sequences of words, such as N-grams and Markov Chains. However, there are also black-box language models that are pretty common in the literature and are proven to yield better accuracy in baseline data sets, such as RNN and C-RNN. This is not to say that they have completely hidden features; much work is put into this recently and the attention mechanism, for example, is a good way to visualize the learned relationships between tokens.

My point really is that GANs need not be so different from what we call a language model. The generator portion of these networks are usually a language model indeed, whether is a standard RNN or a different version of it. The only major difference is the adversarial training with the discriminator as opposed to training based on solely maximum likelihood and categorical cross-entropy. Source

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.