2
$\begingroup$

In the context of GANs I see many papers designing new discriminator networks.

I'm curious about the usefulness of designing discriminators as modified versions of mainstream models like Inception, MobileNet, EfficientNet etc. My intuition is that the mentioned image classification models are way more refined and standardised than a custom discriminator for a specific paper. In addition, I think their pretrained weights should be helpful.

And if they aren't useful, I'm curious why not. Any links about this topic are much appreciated.

$\endgroup$

1 Answer 1

4
$\begingroup$
  • Using a standard network architecture is perfectly reasonable. Most discriminator architectures are trivially different variants of well-known architectures anyway.

  • Depending on the GAN loss, starting with a pretrained network as a discriminator may be dangerous. For instance, the classical GAN loss minimizes the Jensen-Shannon divergence, so having a powerful discriminator right from the start will mean little support overlap and no useful gradients from the discriminator. This is a major reason for training instability in GANs. Yet, more recent losses (e.g., the various WGAN losses) do not have this shared support problem. However, losses like WGAN assume certain properties of the discriminator (i.e., Lipschitz-$k$), which pretrained networks are quite unlikely to satisfy out of the box. So from a purely theoretical perspective, I can see why it's not very appealing.

  • Another reason, I suspect, is that the discriminator's job is quite specialized and requires it to be dynamic; i.e., it has to be able to "change its mind" quickly as the generator adapts. It's not clear to me that a pretrained network, which people have worked to place in a high-quality and robust part of weight space, is good for this. Furthermore, since a discriminator's job is a little easier than e.g. ImageNet classification I suspect that the massive deep networks often used for transfer learning are simply unnecessarily large for the task (the backward or even forward passes being unnecessarily costly, I mean; GANs already take enough time to train).

  • That being said, it is actually common to use pretrained networks for the discriminator for perceptual feature matching losses. (You may be familiar with the even more common "perceptual losses" commonly employed in autoencoders). The idea is to match feature statistics, rather than optimize a scalar probability (this idea comes from Salimans et al, I believe). Note that the pretrained networks used for this (usually VGG) are not trained or even fine-tuned most of the time though; they're usually left alone. Some relevant papers: [1], [2], [3]. I suspect that you can always use a pretrained network in a setup similar to these or in the manner employed by McGAN, and it will be helpful.

  • Despite all this, I am fairly sure that using a pretrained network as a starting point will still work fine in many cases, though I'd suggest starting with a WGAN variant for it to work. It may even do very well, depending on the data and the other training details. Let me know if you try it out =)

$\endgroup$
4
  • $\begingroup$ Thank you for your answer. It was really helpful. I will read your recommended papers. If I try a WGAN variant I'll let you know ;). It sounds promising for what I had in mind. $\endgroup$ Jul 17, 2019 at 15:47
  • $\begingroup$ @SurrealEverything Good luck! $\endgroup$ Jul 17, 2019 at 15:51
  • $\begingroup$ Wow, this is a gem. Thank you for sharing your insights. $\endgroup$ Nov 4, 2020 at 9:58
  • $\begingroup$ datascience.stackexchange.com/questions/94071/… Hi, could you take a look here? $\endgroup$
    – x89
    May 6, 2021 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.