Based on my understanding, gpt or gpt-2 are using language model loss to train and generate text, which do not contains GAN.
So which is better: GPT vs RelGAN/LeakGAN/SeqGAN/TextGAN
I am so confused about this question.
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
According to [Caccia et al., 2018], in general textual GANs are no rival for LMs regarding several quality measures. These are the conclusions of the paper:
This research demonstrates that well-adjusted language models are a remarkably strong baseline and that temperature sweeping can provide a very clear characterization of model performance. A well-adjusted language model outperforms the considered GAN variants as evaluated on both local, and more surprisingly, global metrics of quality and diversity. Our temperature sweeping framework shares characteristics with a Receiver Operating Curve. Analogously, if one needed a single scalar to compare NLG models, one could compute area under the curve and seek the model with the smallest value (lower is better for our considered metrics).
GAN-based generative models have been proven effective on real-valued data, however, but there exist many difficult pernicious issues of moving to discrete data. These issues must be overcome before they will improve over the strong MLE baselines. On the datasets and tasks considered, potential issues caused by exposure bias were less than the issues of training GANs in discrete data. GAN training may prove fruitful eventually, but this research lays forth clear boundaries that it must first surpass.
This way, OpenAI's GPT and GPT-2 may be considered superior in text generation quality to current textual GANs.