6
$\begingroup$

I'm trying to train a DC-GAN on CIFAR-10 Dataset. I'm using Binary Cross Entropy as my loss function for both discriminator and generator (appended with non-trainable discriminator). If I train using Adam optimizer, the GAN is training fine. But if I replace the optimizer by SGD, the training is going haywire. The generator accuracy starts at some higher point and with iterations, it goes to 0 and stays there. The discriminator accuracy starts at some lower point and reaches somewhere around 0.5 (expected, right?). The peculiar thing is the generator loss function is increasing with iterations. I though may be the step is too high. I tried changing the step size. I tried using momentum with SGD. In all these cases, the generator may or may not decrease in the beginning, but then increases for sure. So, I think there is something inherently wrong in my model. I know training Deep Models is difficult and GANs still more, but there has to be some reason/heuristic as to why this is happening. Any inputs in appreciated. I'm new to Neural Networks, Deep Learning and hence new to GANs as well.

Here is my code: Cifar10Models.py

from keras import Sequential
from keras.initializers import TruncatedNormal
from keras.layers import Activation, BatchNormalization, Conv2D, Conv2DTranspose, Dense, Flatten, LeakyReLU, Reshape
from keras.optimizers import SGD


class DcGan:
    def __init__(self, print_model_summary: bool = False):
        self.generator_model = None
        self.discriminator_model = None
        self.concatenated_model = None
        self.print_model_summary = print_model_summary

    def build_generator_model(self):
        if self.generator_model:
            return self.generator_model

        self.generator_model = Sequential()
        self.generator_model.add(Dense(4 * 4 * 512, input_dim=100,
                                       kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
        self.generator_model.add(BatchNormalization(momentum=0.5))
        self.generator_model.add(Activation('relu'))
        self.generator_model.add(Reshape((4, 4, 512)))

        self.generator_model.add(Conv2DTranspose(256, 3, strides=2, padding='same',
                                                 kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
        self.generator_model.add(BatchNormalization(momentum=0.5))
        self.generator_model.add(Activation('relu'))

        self.generator_model.add(Conv2DTranspose(128, 3, strides=2, padding='same',
                                                 kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
        self.generator_model.add(BatchNormalization(momentum=0.5))
        self.generator_model.add(Activation('relu'))

        self.generator_model.add(Conv2DTranspose(64, 3, strides=2, padding='same',
                                                 kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
        self.generator_model.add(BatchNormalization(momentum=0.5))
        self.generator_model.add(Activation('relu'))

        self.generator_model.add(Conv2D(3, 3, padding='same',
                                        kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
        self.generator_model.add(Activation('tanh'))

        if self.print_model_summary:
            self.generator_model.summary()

        return self.generator_model

    def build_discriminator_model(self):
        if self.discriminator_model:
            return self.discriminator_model

        self.discriminator_model = Sequential()
        self.discriminator_model.add(Conv2D(128, 3, strides=2, input_shape=(32, 32, 3), padding='same',
                                            kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
        self.discriminator_model.add(LeakyReLU(alpha=0.2))

        self.discriminator_model.add(Conv2D(256, 3, strides=2, padding='same',
                                            kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
        self.generator_model.add(BatchNormalization(momentum=0.5))
        self.discriminator_model.add(LeakyReLU(alpha=0.2))

        self.discriminator_model.add(Conv2D(512, 3, strides=2, padding='same',
                                            kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
        self.generator_model.add(BatchNormalization(momentum=0.5))
        self.discriminator_model.add(LeakyReLU(alpha=0.2))

        self.discriminator_model.add(Conv2D(1024, 3, strides=2, padding='same',
                                            kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
        self.generator_model.add(BatchNormalization(momentum=0.5))
        self.discriminator_model.add(LeakyReLU(alpha=0.2))

        self.discriminator_model.add(Flatten())
        self.discriminator_model.add(Dense(1, kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))
        self.generator_model.add(BatchNormalization(momentum=0.5))
        self.discriminator_model.add(Activation('sigmoid'))

        if self.print_model_summary:
            self.discriminator_model.summary()

        return self.discriminator_model

    def build_concatenated_model(self):
        if self.concatenated_model:
            return self.concatenated_model

        self.concatenated_model = Sequential()
        self.concatenated_model.add(self.generator_model)
        self.concatenated_model.add(self.discriminator_model)

        if self.print_model_summary:
            self.concatenated_model.summary()

        return self.concatenated_model

    def build_dc_gan(self):
        self.build_generator_model()
        self.build_discriminator_model()
        self.build_concatenated_model()

        self.discriminator_model.trainable = True
        optimizer = SGD(lr=0.0002)
        self.discriminator_model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])
        self.discriminator_model.trainable = False
        optimizer = SGD(lr=0.0001)
        self.concatenated_model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])
        self.discriminator_model.trainable = True

Cifar10Trainer.py:

# Based on https://towardsdatascience.com/gan-by-example-using-keras-on-tensorflow-backend-1a6d515a60d0

import os

import datetime
import numpy
import time
from keras.datasets import cifar10
from keras.utils import np_utils
from matplotlib import pyplot as plt

import Cifar10Models

log_file_name = 'logs.csv'


class Cifar10Trainer:
    def __init__(self):
        self.x_train, self.y_train = self.get_train_and_test_data()
        self.dc_gan = Cifar10Models.DcGan()
        self.dc_gan.build_dc_gan()

    @staticmethod
    def get_train_and_test_data():
        (x_train, y_train), _ = cifar10.load_data()
        x_train = x_train.reshape(x_train.shape[0], x_train.shape[1], x_train.shape[2], 3)
        # Generator output has tanh activation whose range is [-1,1]
        x_train = (x_train.astype('float32') * 2 / 255) - 1
        y_train = np_utils.to_categorical(y_train, 10)
        return x_train, y_train

    def train(self, train_steps=10000, batch_size=128, log_interval=10, save_interval=100,
              output_folder_path='./Trained_Models/'):
        self.initialize_log(output_folder_path)
        self.sample_real_images(output_folder_path)
        for i in range(train_steps):
            # Get real (Database) Images
            images_real = self.x_train[numpy.random.randint(0, self.x_train.shape[0], size=batch_size), :, :, :]

            # Generate Fake Images
            noise = numpy.random.uniform(-1.0, 1.0, size=[batch_size, 100])
            images_fake = self.dc_gan.generator_model.predict(noise)

            # Train discriminator on both real and fake images
            x = numpy.concatenate((images_real, images_fake), axis=0)
            y = numpy.ones([2 * batch_size, 1])
            y[batch_size:, :] = 0
            d_loss = self.dc_gan.discriminator_model.train_on_batch(x, y)

            # Train generator i.e. concatenated model
            noise = numpy.random.uniform(-1.0, 1.0, size=[batch_size, 100])
            y = numpy.ones([batch_size, 1])
            g_loss = self.dc_gan.concatenated_model.train_on_batch(noise, y)

            # Print Logs, Save Models, generate sample images
            if (i + 1) % log_interval == 0:
                self.log_progress(output_folder_path, i + 1, g_loss, d_loss)
            if (i + 1) % save_interval == 0:
                self.save_models(output_folder_path, i + 1)
                self.generate_images(output_folder_path, i + 1)

    @staticmethod
    def initialize_log(output_folder_path):
        log_line = 'Iteration No, Generator Loss, Generator Accuracy, Discriminator Loss, Discriminator Accuracy, ' \
                   'Time\n'
        with open(os.path.join(output_folder_path, log_file_name), 'w') as log_file:
            log_file.write(log_line)

    @staticmethod
    def log_progress(output_folder_path, iteration_no, g_loss, d_loss):
        log_line = '{0:05},{1:2.4f},{2:0.4f},{3:2.4f},{4:0.4f},{5}\n' \
            .format(iteration_no, g_loss[0], g_loss[1], d_loss[0], d_loss[1],
                    datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
        with open(os.path.join(output_folder_path, log_file_name), 'a') as log_file:
            log_file.write(log_line)
        print(log_line)

    def save_models(self, output_folder_path, iteration_no):
        self.dc_gan.generator_model.save(
            os.path.join(output_folder_path, 'generator_model_{0}.h5'.format(iteration_no)))
        self.dc_gan.discriminator_model.save(
            os.path.join(output_folder_path, 'discriminator_model_{0}.h5'.format(iteration_no)))
        self.dc_gan.concatenated_model.save(
            os.path.join(output_folder_path, 'concatenated_model_{0}.h5'.format(iteration_no)))

    def sample_real_images(self, output_folder_path):
        filepath = os.path.join(output_folder_path, 'CIFAR10_Sample_Real_Images.png')
        i = numpy.random.randint(0, self.x_train.shape[0], 16)
        images = self.x_train[i, :, :, :]
        plt.figure(figsize=(10, 10))
        for i in range(16):
            plt.subplot(4, 4, i + 1)
            image = images[i, :, :, :]
            image = numpy.reshape(image, [32, 32, 3])
            plt.imshow(image)
            plt.axis('off')
        plt.tight_layout()
        plt.savefig(filepath)
        plt.close('all')

    def generate_images(self, output_folder_path, iteration_no, noise=None):
        filepath = os.path.join(output_folder_path, 'CIFAR10_Gen_Image{0}.png'.format(iteration_no))
        if noise is None:
            noise = numpy.random.uniform(-1, 1, size=[16, 100])
        # Generator output has tanh activation whose range is [-1,1]
        images = (self.dc_gan.generator_model.predict(noise) + 1) / 2
        plt.figure(figsize=(10, 10))
        for i in range(16):
            plt.subplot(4, 4, i + 1)
            image = images[i, :, :, :]
            image = numpy.reshape(image, [32, 32, 3])
            plt.imshow(image)
            plt.axis('off')
        plt.tight_layout()
        plt.savefig(filepath)
        plt.close('all')


def main():
    cifar10_trainer = Cifar10Trainer()
    cifar10_trainer.train(train_steps=10000, log_interval=10, save_interval=100)
    del cifar10_trainer.dc_gan
    return


if __name__ == '__main__':
    start_time = time.time()
    print('Program Started at {0}'.format(time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(start_time))))
    main()
    end_time = time.time()
    print('Program Ended at {0}'.format(time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(end_time))))
    print('Total Execution Time: {0}s'.format(datetime.timedelta(seconds=end_time - start_time)))

Some of the graphs are as below:

  1. Discriminator Optimizer: Adam(lr=0.0001, beta1=0.5)
    Generator Optimizer: Adam(lr=0.0001, beta1=0.5) enter image description here

  2. Discriminator Optimizer: SGD(lr=0.0001)
    Generator Optimizer: SGD(lr=0.0001) enter image description here

  3. Discriminator Optimizer: SGD(lr=0.0001)
    Generator Optimizer: SGD(lr=0.001) enter image description here

  4. Discriminator Optimizer: SGD(lr=0.0001)
    Generator Optimizer: SGD(lr=0.0005) enter image description here

Note:
This question was originally asked in StackOverflow and then re-asked here as per suggestions in SO

Edit1:
Adding some generated images for reference

  1. Images generated by Adam Optimizer enter image description here

  2. Images generated by SGD Optimizer enter image description here

$\endgroup$
1
  • $\begingroup$ Your Adam optimizer params a bit different than the original paper. $\endgroup$ Nov 2, 2021 at 0:40

1 Answer 1

9
$\begingroup$

I think that there are several issues with your model:

First of all - Your generator's loss is not the generator's loss. You have on binary cross-entropy loss function for the discriminator, and you have another binary cross-entropy loss function for the concatenated model whose output is again the discriminator's output (on generated images). The "generator loss" you are showing is the discriminator's loss when dealing with generated images. You want this loss to go up, it means that your model successfully generates images that you discriminator fails to catch (as can be seen in the overall discriminator's accuracy which is at 0.5).

Another issue, is that you should add some generator regularization in the form of an actual generator loss ("generator objective function"). You can read about the different options in GAN Objective Functions: GANs and Their Variations.

A final issue that I see is that you are passing the generated images thru a final hyperbolic tangent activation function, and I don't really understand why? The generator in your case is supposed to generate a "believable" CIFAR10 image, which is a 32x32x3 tensor with values in the range [0,255] or [0,1]. Your generator's output has a potential range of [-1,1] (as you state in your code).

$\endgroup$
7
  • $\begingroup$ Happy 1K! Comments must be at least 15 characters in length. $\endgroup$
    – Esmailian
    Mar 24, 2019 at 8:26
  • 1
    $\begingroup$ Thank you very much :) $\endgroup$
    – Mark.F
    Mar 24, 2019 at 10:15
  • 1
    $\begingroup$ Thanks. 1. What I've defined as generator_loss, it is the binary cross entropy between the discriminator output and the desired output, which is 1 while training generator. Now, if my generator is able to fool the discriminator, then discriminator output should be close to 1, right?. So, the bce value should decrease. Right? Also, if you see the first graph where I've used Adam instead of SGD, the loss didn't increase. In that case, the generated images are better. When using SGD, the generated images are noise. Since generator accuracy is 0, the discriminator accuracy of 0.5 doesn't mean much $\endgroup$ Mar 24, 2019 at 11:54
  • $\begingroup$ 2. I'll look into GAN objective functions. I was trying to implement plain DCGAN paper. 3. I'm using tanh function because DC-GAN paper says so. Yes, even though tanh outputs in the range [-1,1], if you see the generate_images function in Trainer.py file, I'm doing this: images = (self.dc_gan.generator_model.predict(noise) + 1) / 2 So, this is a valid Image right? Also, like the first graph showed, even with this, using Adam Optimizer, I'm getting good results. Using SGD is causes this problem. The only change is SGD is used instead of Adam. So, there must be some problem with this. $\endgroup$ Mar 24, 2019 at 11:58
  • 1
    $\begingroup$ In the original DCGan paper the images are also normalized to [-1,1], and this is standard practice in training networks on images. $\endgroup$ Nov 2, 2021 at 0:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.