Why is my DCGAN not converging?

Question

I'm training a tf DCGAN on the MVTec hazelnut dataset and I found some difficulties. The problem is that after a lot of epochs the generate does not produce some quality images.

My model is the following:

    def build_generator(self, noise_dim):
        #
        # Input shape (None, 100)
        # Output shape (None, 64, 64, 3)
        #
        model = Sequential(name = "Generator")

        model.add(Dense(4*4*512, input_dim = noise_dim, activation = "relu", use_bias = False))
        model.add(BatchNormalization())
        model.add(Reshape((4,4,512)))

        model.add(Conv2DTranspose(512, (5, 5), strides = (2, 2), padding = "same", activation = "relu", use_bias = False))
        model.add(BatchNormalization())

        model.add(Conv2DTranspose(256, (5, 5), strides = (2, 2), padding = "same", activation = "relu", use_bias = False))
        model.add(BatchNormalization())

        model.add(Conv2DTranspose(128, (5, 5), strides = (2, 2), padding = "same", activation = "relu", use_bias = False))
        model.add(BatchNormalization())

        model.add(Conv2DTranspose(64, (5, 5), strides = (2, 2), padding = "same", activation = "relu", use_bias = False))
        model.add(BatchNormalization())

        model.add(Conv2DTranspose(3, (5, 5), activation = "tanh", padding = "same", use_bias = False))

        assert model.output_shape == (None, 64, 64, 3)

        return model

    def build_discriminator(self, image_shape):

        model = Sequential(name = "Discriminator")

        model.add(Conv2D(64, kernel_size = (5, 5), strides = (2, 2), input_shape = image_shape, padding = "same"))
        model.add(BatchNormalization())
        model.add(LeakyReLU(0.2))
        model.add(Dropout(0.3))

        model.add(Conv2D(128, kernel_size = (5, 5), strides = (2, 2), padding = "same"))
        model.add(BatchNormalization())
        model.add(LeakyReLU(0.2))
        model.add(Dropout(0.3))

        model.add(Conv2D(256, kernel_size = (5, 5), strides = (2, 2), padding = "same"))
        model.add(BatchNormalization())
        model.add(LeakyReLU(0.2))
        model.add(Dropout(0.3))

        model.add(Conv2D(512, kernel_size = (5, 5), strides = (2, 2), padding = "same"))
        model.add(BatchNormalization())
        model.add(LeakyReLU(0.2))
        model.add(Dropout(0.3))

        model.add(Flatten())
        model.add(Dense(1))

        return model

I tried a few configurations of the previous one and i think this is the best and it seems to learn faster than models with less filters.

About the training set, it has only 391 images and it seems to be a very poor dataset to train a GAN, so I tried some data augmentation to gather more images. In particular I've used the RandomTranslation, RandomRotation and RandomZoom tensorflow's layers. After that with 1564 images (normalized betweem -1 and 1) the training seems, again, a little faster: after 100 epochs the G_loss has a value between 1 and 2 and the D_loss is between 0.5 and 1. However, the generated images don't seems real.

Can anyone help out on how to make my GAN to converge? Thanks in advance.

Answer 1

First, your discriminator output should use a sigmoid activation (assuming you use the binary crossentropy or wasserstein loss). I am unsure if this will fix your problem though. I think the reason your model doesn't converge is the small number of samples you use for training compared to the relatively large complexity of your model. You could try the same architecture with MNIST or the CelebA data sets (70.000 and ~200.000 images) and see if you still have the issue.

Answer 2

I post my solution to this question in case someone else is having a similar issue.

In this case the GAN was not converging because the discriminator was too good. This is a common problem: if discriminator can distinguish the real examples from the fake ones it will never give any suggestion to the generator how to improve the data generation. So the model diverges.

In this case i removed one hidden layer from the net.

def build_discriminator(self):

        model = Sequential(name = "Discriminator")

        model.add(Conv2D(64, kernel_size = (5, 5), strides = (2, 2), input_shape = self.image_shape, padding = "same"))
        model.add(BatchNormalization())
        model.add(LeakyReLU(0.2))
        model.add(Dropout(0.3))

        model.add(Conv2D(128, kernel_size = (5, 5), strides = (2, 2), padding = "same"))
        model.add(BatchNormalization())
        model.add(LeakyReLU(0.2))
        model.add(Dropout(0.3))

        model.add(Conv2D(256, kernel_size = (5, 5), strides = (2, 2), padding = "same"))
        model.add(BatchNormalization())
        model.add(LeakyReLU(0.2))
        model.add(Dropout(0.3))

        model.add(Flatten())
        model.add(Dense(1))

        return model