GANs (generative adversarial networks) possible for text as well?

Question

Are GANs - generative adversarial networks - good just for images, or could be used for text as well?

Like, train a network to generate meaningful texts from a summary.

UPD - quotes from the GAN inventor Ian Goodfellow.

GANs have not been applied to NLP because GANs are only defined for real-valued data. (2016) source

It is not a fundamentally flawed idea. It should be possible to do at least one of the following... (2017) source

Answer 1

Yes, GANs can be used for text. However, there is a problem in the combination of how GANs work and how text is normally generated by neural networks:

• GANs work by propagating gradients through the composition of Generator and Discriminator.
• Text is normally generated by having a final softmax layer over the token space, that is, the output of the network is normally the probabilities of generating each token (i.e. a discrete stochastic unit).

These 2 things do not work well together on their own, because you cannot propagate gradients through discrete stochastic units. There are 2 main approaches to deal with this: the REINFORCE algorithm and the Gumbel-Softmax reparameterization (also known as the Concrete distribution). Take into account that REINFORCE is known to have high variance so you need large amounts of data to get good gradient estimations.

As an example of REINFORCE for textual GANs you can check the SeqGAN article. An example of Gumbel-Softmax you can check this article.

Another completely different option is not having a discrete stochastic unit as output of the generator (e.g. generating tokens deterministically in embedded space), hence eliminating the original problem of backpropagating through them.

Answer 2

There is even more specific research on this topic:

The trained generator is capable of producing sentences with certain level of grammar and logic.

Xuerong Xiao, "Text Generation using Generative Adversarial Training"

This question relates to this one: https://linguistics.stackexchange.com/questions/26448/how-to-translate-pelevins-creative-unit-idea-to-a-scientific-problem