Are mainstream pre-trained models useful as discriminators?

Question

In the context of GANs I see many papers designing new discriminator networks.

I'm curious about the usefulness of designing discriminators as modified versions of mainstream models like Inception, MobileNet, EfficientNet etc. My intuition is that the mentioned image classification models are way more refined and standardised than a custom discriminator for a specific paper. In addition, I think their pretrained weights should be helpful.

And if they aren't useful, I'm curious why not. Any links about this topic are much appreciated.

Answer 1

• Using a standard network architecture is perfectly reasonable. Most discriminator architectures are trivially different variants of well-known architectures anyway.

• Depending on the GAN loss, starting with a pretrained network as a discriminator may be dangerous. For instance, the classical GAN loss minimizes the Jensen-Shannon divergence, so having a powerful discriminator right from the start will mean little support overlap and no useful gradients from the discriminator. This is a major reason for training instability in GANs. Yet, more recent losses (e.g., the various WGAN losses) do not have this shared support problem. However, losses like WGAN assume certain properties of the discriminator (i.e., Lipschitz-$k$), which pretrained networks are quite unlikely to satisfy out of the box. So from a purely theoretical perspective, I can see why it's not very appealing.

• Another reason, I suspect, is that the discriminator's job is quite specialized and requires it to be dynamic; i.e., it has to be able to "change its mind" quickly as the generator adapts. It's not clear to me that a pretrained network, which people have worked to place in a high-quality and robust part of weight space, is good for this. Furthermore, since a discriminator's job is a little easier than e.g. ImageNet classification I suspect that the massive deep networks often used for transfer learning are simply unnecessarily large for the task (the backward or even forward passes being unnecessarily costly, I mean; GANs already take enough time to train).

• That being said, it is actually common to use pretrained networks for the discriminator for perceptual feature matching losses. (You may be familiar with the even more common "perceptual losses" commonly employed in autoencoders). The idea is to match feature statistics, rather than optimize a scalar probability (this idea comes from Salimans et al, I believe). Note that the pretrained networks used for this (usually VGG) are not trained or even fine-tuned most of the time though; they're usually left alone. Some relevant papers: [1], [2], [3]. I suspect that you can always use a pretrained network in a setup similar to these or in the manner employed by McGAN, and it will be helpful.

• Despite all this, I am fairly sure that using a pretrained network as a starting point will still work fine in many cases, though I'd suggest starting with a WGAN variant for it to work. It may even do very well, depending on the data and the other training details. Let me know if you try it out =)