Strange patterns from GAN

Question

I'm doing experiments with GAN. I've successfully trained GAN on 28x28px MNIST dataset (samples scaled to (-1,1) interval)

My next experiment is to train GAN on bigger images. My dataset consists of grayscale 128x128px images. I've added one more convolutional layer to both generator and discriminator and updated dense layers accordingly.

Sample from MNIST dataset and sample from my custom dataset:

However, when I trained the model, all results from generator looks like this:

What am I doing wrong? Why does the generator generate only this noise pattern instead of samples from my dataset distribution?

Generator:

H = Sequential()

H.add(Dense(1024, input_shape=(noise_dim,)))
H.add(BatchNormalization())
H.add(Activation('relu'))

H.add(Dense(128*16*16))
H.add(BatchNormalization())
H.add(Activation('relu'))
H.add(Reshape([16, 16, 128]))

H.add(Conv2DTranspose(128, (3, 3), strides=(2, 2), padding='same'))
H.add(BatchNormalization())
H.add(Activation('relu'))

H.add(Conv2DTranspose(64, (3, 3), strides=(2, 2), padding='same'))
H.add(BatchNormalization())
H.add(Activation('relu'))

H.add(Conv2DTranspose(32, (3, 3), strides=(2, 2), padding='same'))
H.add(BatchNormalization())
H.add(Activation('relu'))

H.add(Conv2DTranspose(1, (3, 3), padding='same'))
H.add(Activation('tanh', name='out_image'))

gen_in = Input(shape=(noise_dim,), name='in_noise')
gen_out = H(gen_in)

Discriminator:

H = Sequential()

H.add(Conv2D(32, (3, 3), strides=(2, 2), padding='same', activation='relu', input_shape=(128, 128, 1)))
H.add(LeakyReLU(0.2))

H.add(Conv2D(64, (3, 3), strides=(2, 2), padding='same', activation='relu'))
H.add(BatchNormalization())
H.add(LeakyReLU(0.2))

H.add(Conv2D(128, (3, 3), strides=(2, 2), padding='same', activation='relu'))
H.add(BatchNormalization())
H.add(LeakyReLU(0.2))
H.add(Flatten())

H.add(Dense(1024))
H.add(BatchNormalization())
H.add(LeakyReLU(0.2))

H.add(Dense(1, activation='sigmoid', name='out_binary'))

disc_in = Input(shape=(128, 128, 1), name='in_image')
disc_out = H(disc_in)

Answer 1

The deconv layers are probably to blame. Check out this distill article for a fairly in depth discussion about how deconv layers create checkerboard artifacts. The gist is that deconv striding creates interference patterns which can cancel out if you're careful, but are more likely to be worsened as you add more deconv layers. The authors suggest using nearest neighbors for upsampling followed by a convolutional layer, which they call resize-convolution upsampling.

Answer 2

Training GANs becomes much more difficult with increasing input dimensionality. I encountered that sometimes even small changes to your network can decide about convergence/divergence of your network (like initialization of your variables).

I recommend following the tipps from https://github.com/soumith/ganhacks.

Another thing I personally recognized is that you should not use as many Conv2DTranspose layers as you do in your current network. Try instead increasing the latent space.