up vote 13 down vote favorite
7

Are GANs - generative adversarial networks - good just for images, or could be used for text as well?

Like, train a network to generate meaningful texts from a summary.

UPD - quotes from the GAN inventor Ian Goodfellow.

GANs have not been applied to NLP because GANs are only defined for real-valued data. (2016) source

It is not a fundamentally flawed idea. It should be possible to do at least one of the following... (2017) source

share|improve this question
  • 2
    The quote you cite is from January 2016, so not very up to date. Here's a more recent answer (December 2016) also by Ian Goodfellow about the same topic, where he mentions a few ways of using GANs with text. – ncasas Nov 23 '17 at 10:52
up vote 13 down vote accepted

Yes, GANs can be used for text. However, there is a problem in the combination of how GANs work and how text is normally generated by neural networks:

  • GANs work by propagating gradients through the composition of Generator and Discriminator.
  • Text is normally generated by having a final softmax layer over the token space, that is, the output of the network is normally the probabilities of generating each token (i.e. a discrete stochastic unit).

These 2 things do not work well together on their own, because you cannot propagate gradients through discrete stochastic units. There are 2 main approaches to deal with this: the REINFORCE algorithm and the Gumbel-Softmax reparameterization (also known as the Concrete distribution). Take into account that REINFORCE is known to have high variance so you need large amounts of data to get good gradient estimations.

As an example of REINFORCE for textual GANs you can check the SeqGAN article. An example of Gumbel-Softmax you can check this article.

Another completely different option is not having a discrete stochastic unit as output of the generator (e.g. generating tokens deterministically in embedded space), hence eliminating the original problem of backpropagating through them.

share|improve this answer
up vote 3 down vote

There is even more specific research on this topic:

The trained generator is capable of producing sentences with certain level of grammar and logic.

Xuerong Xiao, "Text Generation using Generative Adversarial Training"

This question relates to this one: https://linguistics.stackexchange.com/questions/26448/how-to-translate-pelevins-creative-unit-idea-to-a-scientific-problem

share|improve this answer

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.