In my GAN model, the discriminator loss quickly descends to magnitudes of $10^{-4}$ while generator loss is at levels of 5+?

Question

I am creating a Generative Adversarial Network (GAN) for generating artificial trading cards, but I am a complete novice in the field. The problem I'm consistently having is that my discriminator, even though it is weaker (on the basis of learnable parameters), has its loss drop to magnitudes of $10^{-4}$ (ten to the power of negative four). In contrast, the generator loss accelerates from 5+ to 10+ in the first few epochs. Further, the discriminator's accuracy on both real and fake images instantly goes to 100%, varying at most by 2% on either end.

My current generative model:

def generator_model():

    model = tf.keras.Sequential()

    # First Dense Layer
    model.add(Dense(8*8*64, input_dim=100)) #input_shape=(100,)))
    model.add(BatchNormalization())
    model.add(LeakyReLU())

    model.add(Reshape((8, 8, 64)))

    # First Conv2DTranspose Layer
    model.add(Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False))
    model.add(LeakyReLU())

    # Second Conv2DTranspose Layer
    model.add(Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))
    model.add(LeakyReLU())

    # Third Conv2DTranspose Layer
    model.add(Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False))
    model.add(LeakyReLU())

    # Fourth Conv2DTranspose Layer
    model.add(Conv2DTranspose(32, (5, 5), strides=(2, 2), padding='same', use_bias=False))
    model.add(LeakyReLU())

    # Fifth Conv2DTranspose Layer
    model.add(Conv2DTranspose(3, (3, 3), strides=(1, 1), padding='same', use_bias=False, activation='tanh'))

    return model

My current discriminator model:

def discriminator_model():

    model = tf.keras.Sequential()

    # First Conv2D Layer
    model.add(Conv2D(64, (5, 5), strides=(1, 1), padding='same', input_shape=[64, 64, 3]))
    model.add(LeakyReLU())
    model.add(Dropout(0.3))

    # Second Conv2D Layer
    model.add(Conv2D(64, (5, 5), strides=(2, 2), padding='same'))
    model.add(LeakyReLU())
    model.add(Dropout(0.3))

    # Third Conv2D Layer
    model.add(Conv2D(64, (5, 5), strides=(2, 2), padding='same'))
    model.add(LeakyReLU())
    model.add(Dropout(0.3))

    # Fourth Conv2D Layer
    model.add(Conv2D(32, (5, 5), strides=(2, 2), padding='same'))
    model.add(LeakyReLU())
    model.add(Dropout(0.3))

    # Flatten the Output and Give Binary Output via Sigmoid Activation Function
    model.add(Flatten())
    model.add(Dense(1, activation='sigmoid'))

    optimizer = tf.keras.optimizers.Adam(learning_rate=2e-4, beta_1=0.5)

    # Compile the Discriminator Model
    model.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])

    return model

My current GAN model:

def gan_model(generator, discriminator):
    GAN = tf.keras.Sequential()
    discriminator.trainable = False

    GAN.add(generator)
    GAN.add(discriminator)

    optimizer = tf.keras.optimizers.Adam(learning_rate=2e-4, beta_1=0.5)

    GAN.compile(loss='binary_crossentropy', optimizer=optimizer)

    return GAN

My current GAN training method: (I'm wondering if this is a terribly constructed training step)

def training_gan(gan_model, discriminator, generator, batch_size=256, epochs=100, epoch_steps=468, noise_dim=100):
    # Training the model by enumerating epochs 
    for epoch in range(0,epochs): 
        for step in range(0, epoch_steps):
            # Generating fake images 
            X_fake, y_fake = generate_img_using_model(generator, noise_dim, batch_size)
            # Generating real images 
            X_real, y_real = generate_real_images(batch_size)
            # Creating training set
            X_batch = np.concatenate([X_real, X_fake], axis = 0)
            y_batch = np.concatenate([y_real, y_fake], axis = 0)      
            # Training the discriminator
            d_loss, _ = discriminator.train_on_batch(X_batch, y_batch)
            # Gnerating noise input for the generator 
            X_gan = np.random.randn(noise_dim * batch_size)
            X_gan = X_gan.reshape(batch_size, noise_dim)
            y_gan = np.ones((batch_size, 1))
            # Training the GAN model using the generated noise 
            gan_loss = gan_model.train_on_batch(X_gan,y_gan)
            # Report the training progress
            areport_progress(epoch=epoch, step=step, d_loss=d_loss, gan_loss=gan_loss, noise_dim=noise_dim, epoch_steps=epoch_steps)
    # Report the progress on the full epoch
    report_progress(epoch=epoch, step=step, d_loss=d_loss, gan_loss=gan_loss, noise_dim=noise_dim, epoch_steps=epoch_steps, gan_model=gan_model, generator=generator, discriminator=discriminator, eoe=True)

My current 'report_progress' method:

def report_progress(epoch, step, d_loss, gan_loss, noise_dim = None, epoch_steps= None, gan_model=None, generator=None, discriminator=None, n_samples=100, eoe= False):
    if eoe and step == (epoch_steps-1):
        # Report a full epoch training performance
        # Sample some real images from the training set
        X_real, y_real = generate_real_images(n_samples)
        # Measure the accuracy of the discrinminator on real sampled images
        _ , acc_real = discriminator.evaluate(X_real, y_real, verbose=0)
        # Generates fake examples
        X_fake, y_fake = generate_img_using_model(generator, noise_dim, n_samples)
        # evaluate discriminator on fake images
        _, acc_fake = discriminator.evaluate(X_fake, y_fake, verbose=0)
        # summarize discriminator performance
        # plot images
        plt.figure(figsize=(20, 12), dpi=64)
        for i in range(10 * 10):
            # define subplot
            plt.subplot(10, 10, 1 + I)
            # turn off axis
            plt.axis('off')
            # plot raw pixel data
            plt.imshow(upscale_image(X_fake[i, :, :, :])) #, cmap='gray_r')
            #plt.show()
        filename = 'generated_examples_epoch%04d.png' % (epoch+1)
        plt.savefig(filename)
        print('Disciminator Accuracy on real images: %.0f%%, on fake images: %.0f%%' % (acc_real*100, acc_fake*100))
        # save the generator model tile file
        filename = 'generator_epochs/generator_model_%04d.h5' % epoch
        generator.save(filename)
        filename = 'discriminator_epochs/discriminator_model_%04d.h5' % epoch
        discriminator.save(filename)
        filename = 'GAN_epochs/GAN_model_%04d.h5' % epoch
        gan_model.save(filename)
    elif step % 10 == 0:
        # Report a single step training performance 
        print(f"[Epoch {epoch}, Step {step}] d_loss = {round(d_loss, 4)} | gan_loss = {round(gan_loss, 4)}")

Note: All of the above code is drawn mostly from a tutorial that I no longer have the link to readily available.

Additional Questions:

• Why, in so many tutorials, is 'use_bias' set to 'False' in all 'Conv2DTranspose' layers?
• Why is the 'tanh' activator used in the final 'Conv2DTranspose' layer?
• Why is 'discriminator.trainable' set to 'False' in the GAN model?

Any advice and/or recommended readings on the basics of building synergistic generative and discriminator networks would be much appreciated as well.

Answer 1

I can give you an answer regarding two of the three additional question. The reason that the activation for the last convolutional tranpose layer is set to tanh is most likely because the input data is also preprocessed to be in the (-1, 1) range, so it's making sure that both the real images and the generated images have a similar distribution. If the activation for the last layer would be different (e.g. a sigmoid activation), the ranges of the two distributions would differ quite a lot making it easy for the discriminator to determine if an image is real or fake.

Regarding the second question of setting the trainable attribute to False for the discriminator in the gan_model function, this seems to be related to allowing the discriminator to train when discriminator.train_on_batch is called but not when gan.train_on_batch for the combined model (see also this answer on a github issue)