Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
Are GANs - generative adversarial networks - good just for images, or could be used for text as well?
Like, train a network to generate meaningful texts from a summary.
UPD - quotes from the GAN inventor Ian Goodfellow.
GANs have not been applied to NLP because GANs are only defined for real-valued data. (2016) source
It is not a fundamentally flawed idea. It should be possible to do at least one of the following... (2017) source
Yes, GANs can be used for text. However, there is a problem in the combination of how GANs work and how text is normally generated by neural networks:
These 2 things do not work well together on their own, because you cannot propagate gradients through discrete stochastic units. There are 2 main approaches to deal with this: the REINFORCE algorithm and the Gumbel-Softmax reparameterization (also known as the Concrete distribution). Take into account that REINFORCE is known to have high variance so you need large amounts of data to get good gradient estimations.
As an example of REINFORCE for textual GANs you can check the SeqGAN article. An example of Gumbel-Softmax you can check this article.
Another completely different option is not having a discrete stochastic unit as output of the generator (e.g. generating tokens deterministically in embedded space), hence eliminating the original problem of backpropagating through them.
There is even more specific research on this topic:
The trained generator is capable of producing sentences with certain level of grammar and logic.
Xuerong Xiao, "Text Generation using Generative Adversarial Training"
This question relates to this one: https://linguistics.stackexchange.com/questions/26448/how-to-translate-pelevins-creative-unit-idea-to-a-scientific-problem
Yes, GANs can now be used for discrete data as well. The first instance of this intuition came when Wasserstein GANs (WGAN) came into existence. Ian Goodfellow addressed a Reinforcement Learning approach to this problem in the NIPS 2016 Conference Also, This article deals with advancements in GAN with respect to discrete data.
