# Why is my generator loss function increasing with iterations?

I'm trying to train a DC-GAN on CIFAR-10 Dataset. I'm using Binary Cross Entropy as my loss function for both discriminator and generator (appended with non-trainable discriminator). If I train using Adam optimizer, the GAN is training fine. But if I replace the optimizer by SGD, the training is going haywire. The generator accuracy starts at some higher point and with iterations, it goes to 0 and stays there. The discriminator accuracy starts at some lower point and reaches somewhere around 0.5 (expected, right?). The peculiar thing is the generator loss function is increasing with iterations. I though may be the step is too high. I tried changing the step size. I tried using momentum with SGD. In all these cases, the generator may or may not decrease in the beginning, but then increases for sure. So, I think there is something inherently wrong in my model. I know training Deep Models is difficult and GANs still more, but there has to be some reason/heuristic as to why this is happening. Any inputs in appreciated. I'm new to Neural Networks, Deep Learning and hence new to GANs as well.

Here is my code: Cifar10Models.py

from keras import Sequential
from keras.initializers import TruncatedNormal
from keras.layers import Activation, BatchNormalization, Conv2D, Conv2DTranspose, Dense, Flatten, LeakyReLU, Reshape
from keras.optimizers import SGD

class DcGan:
def __init__(self, print_model_summary: bool = False):
self.generator_model = None
self.discriminator_model = None
self.concatenated_model = None
self.print_model_summary = print_model_summary

def build_generator_model(self):
if self.generator_model:
return self.generator_model

self.generator_model = Sequential()
self.generator_model.add(Dense(4 * 4 * 512, input_dim=100,
kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))

kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))

kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))

kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))

kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))

if self.print_model_summary:
self.generator_model.summary()

return self.generator_model

def build_discriminator_model(self):
if self.discriminator_model:
return self.discriminator_model

self.discriminator_model = Sequential()
kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))

kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))

kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))

kernel_initializer=TruncatedNormal(mean=0.0, stddev=0.02)))

if self.print_model_summary:
self.discriminator_model.summary()

return self.discriminator_model

def build_concatenated_model(self):
if self.concatenated_model:
return self.concatenated_model

self.concatenated_model = Sequential()

if self.print_model_summary:
self.concatenated_model.summary()

return self.concatenated_model

def build_dc_gan(self):
self.build_generator_model()
self.build_discriminator_model()
self.build_concatenated_model()

self.discriminator_model.trainable = True
optimizer = SGD(lr=0.0002)
self.discriminator_model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])
self.discriminator_model.trainable = False
optimizer = SGD(lr=0.0001)
self.concatenated_model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])
self.discriminator_model.trainable = True


Cifar10Trainer.py:

# Based on https://towardsdatascience.com/gan-by-example-using-keras-on-tensorflow-backend-1a6d515a60d0

import os

import datetime
import numpy
import time
from keras.datasets import cifar10
from keras.utils import np_utils
from matplotlib import pyplot as plt

import Cifar10Models

log_file_name = 'logs.csv'

class Cifar10Trainer:
def __init__(self):
self.x_train, self.y_train = self.get_train_and_test_data()
self.dc_gan = Cifar10Models.DcGan()
self.dc_gan.build_dc_gan()

@staticmethod
def get_train_and_test_data():
x_train = x_train.reshape(x_train.shape[0], x_train.shape[1], x_train.shape[2], 3)
# Generator output has tanh activation whose range is [-1,1]
x_train = (x_train.astype('float32') * 2 / 255) - 1
y_train = np_utils.to_categorical(y_train, 10)
return x_train, y_train

def train(self, train_steps=10000, batch_size=128, log_interval=10, save_interval=100,
output_folder_path='./Trained_Models/'):
self.initialize_log(output_folder_path)
self.sample_real_images(output_folder_path)
for i in range(train_steps):
# Get real (Database) Images
images_real = self.x_train[numpy.random.randint(0, self.x_train.shape[0], size=batch_size), :, :, :]

# Generate Fake Images
noise = numpy.random.uniform(-1.0, 1.0, size=[batch_size, 100])
images_fake = self.dc_gan.generator_model.predict(noise)

# Train discriminator on both real and fake images
x = numpy.concatenate((images_real, images_fake), axis=0)
y = numpy.ones([2 * batch_size, 1])
y[batch_size:, :] = 0
d_loss = self.dc_gan.discriminator_model.train_on_batch(x, y)

# Train generator i.e. concatenated model
noise = numpy.random.uniform(-1.0, 1.0, size=[batch_size, 100])
y = numpy.ones([batch_size, 1])
g_loss = self.dc_gan.concatenated_model.train_on_batch(noise, y)

# Print Logs, Save Models, generate sample images
if (i + 1) % log_interval == 0:
self.log_progress(output_folder_path, i + 1, g_loss, d_loss)
if (i + 1) % save_interval == 0:
self.save_models(output_folder_path, i + 1)
self.generate_images(output_folder_path, i + 1)

@staticmethod
def initialize_log(output_folder_path):
log_line = 'Iteration No, Generator Loss, Generator Accuracy, Discriminator Loss, Discriminator Accuracy, ' \
'Time\n'
with open(os.path.join(output_folder_path, log_file_name), 'w') as log_file:
log_file.write(log_line)

@staticmethod
def log_progress(output_folder_path, iteration_no, g_loss, d_loss):
log_line = '{0:05},{1:2.4f},{2:0.4f},{3:2.4f},{4:0.4f},{5}\n' \
.format(iteration_no, g_loss[0], g_loss[1], d_loss[0], d_loss[1],
datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S'))
with open(os.path.join(output_folder_path, log_file_name), 'a') as log_file:
log_file.write(log_line)
print(log_line)

def save_models(self, output_folder_path, iteration_no):
self.dc_gan.generator_model.save(
os.path.join(output_folder_path, 'generator_model_{0}.h5'.format(iteration_no)))
self.dc_gan.discriminator_model.save(
os.path.join(output_folder_path, 'discriminator_model_{0}.h5'.format(iteration_no)))
self.dc_gan.concatenated_model.save(
os.path.join(output_folder_path, 'concatenated_model_{0}.h5'.format(iteration_no)))

def sample_real_images(self, output_folder_path):
filepath = os.path.join(output_folder_path, 'CIFAR10_Sample_Real_Images.png')
i = numpy.random.randint(0, self.x_train.shape[0], 16)
images = self.x_train[i, :, :, :]
plt.figure(figsize=(10, 10))
for i in range(16):
plt.subplot(4, 4, i + 1)
image = images[i, :, :, :]
image = numpy.reshape(image, [32, 32, 3])
plt.imshow(image)
plt.axis('off')
plt.tight_layout()
plt.savefig(filepath)
plt.close('all')

def generate_images(self, output_folder_path, iteration_no, noise=None):
filepath = os.path.join(output_folder_path, 'CIFAR10_Gen_Image{0}.png'.format(iteration_no))
if noise is None:
noise = numpy.random.uniform(-1, 1, size=[16, 100])
# Generator output has tanh activation whose range is [-1,1]
images = (self.dc_gan.generator_model.predict(noise) + 1) / 2
plt.figure(figsize=(10, 10))
for i in range(16):
plt.subplot(4, 4, i + 1)
image = images[i, :, :, :]
image = numpy.reshape(image, [32, 32, 3])
plt.imshow(image)
plt.axis('off')
plt.tight_layout()
plt.savefig(filepath)
plt.close('all')

def main():
cifar10_trainer = Cifar10Trainer()
cifar10_trainer.train(train_steps=10000, log_interval=10, save_interval=100)
del cifar10_trainer.dc_gan
return

if __name__ == '__main__':
start_time = time.time()
print('Program Started at {0}'.format(time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(start_time))))
main()
end_time = time.time()
print('Program Ended at {0}'.format(time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(end_time))))
print('Total Execution Time: {0}s'.format(datetime.timedelta(seconds=end_time - start_time)))


Some of the graphs are as below:

2. Discriminator Optimizer: SGD(lr=0.0001)
Generator Optimizer: SGD(lr=0.0001)

3. Discriminator Optimizer: SGD(lr=0.0001)
Generator Optimizer: SGD(lr=0.001)

4. Discriminator Optimizer: SGD(lr=0.0001)
Generator Optimizer: SGD(lr=0.0005)

Note:
This question was originally asked in StackOverflow and then re-asked here as per suggestions in SO

Edit1:
Adding some generated images for reference

1. Images generated by Adam Optimizer

2. Images generated by SGD Optimizer

I think that there are several issues with your model:

First of all - Your generator's loss is not the generator's loss. You have on binary cross-entropy loss function for the discriminator, and you have another binary cross-entropy loss function for the concatenated model whose output is again the discriminator's output (on generated images). The "generator loss" you are showing is the discriminator's loss when dealing with generated images. You want this loss to go up, it means that your model successfully generates images that you discriminator fails to catch (as can be seen in the overall discriminator's accuracy which is at 0.5).

Another issue, is that you should add some generator regularization in the form of an actual generator loss ("generator objective function"). You can read about the different options in GAN Objective Functions: GANs and Their Variations.

A final issue that I see is that you are passing the generated images thru a final hyperbolic tangent activation function, and I don't really understand why? The generator in your case is supposed to generate a "believable" CIFAR10 image, which is a 32x32x3 tensor with values in the range [0,255] or [0,1]. Your generator's output has a potential range of [-1,1] (as you state in your code).

• Happy 1K! Comments must be at least 15 characters in length. – Esmailian Mar 24 '19 at 8:26
• Thank you very much :) – Mark.F Mar 24 '19 at 10:15
• Thanks. 1. What I've defined as generator_loss, it is the binary cross entropy between the discriminator output and the desired output, which is 1 while training generator. Now, if my generator is able to fool the discriminator, then discriminator output should be close to 1, right?. So, the bce value should decrease. Right? Also, if you see the first graph where I've used Adam instead of SGD, the loss didn't increase. In that case, the generated images are better. When using SGD, the generated images are noise. Since generator accuracy is 0, the discriminator accuracy of 0.5 doesn't mean much – Nagabhushan S N Mar 24 '19 at 11:54
• 2. I'll look into GAN objective functions. I was trying to implement plain DCGAN paper. 3. I'm using tanh function because DC-GAN paper says so. Yes, even though tanh outputs in the range [-1,1], if you see the generate_images function in Trainer.py file, I'm doing this: images = (self.dc_gan.generator_model.predict(noise) + 1) / 2 So, this is a valid Image right? Also, like the first graph showed, even with this, using Adam Optimizer, I'm getting good results. Using SGD is causes this problem. The only change is SGD is used instead of Adam. So, there must be some problem with this. – Nagabhushan S N Mar 24 '19 at 11:58
• I've added some generated images for reference. Please check them as well. Again, thanks a lot for your time and suggestions – Nagabhushan S N Mar 24 '19 at 12:02