Currently training a WGAN with weight clipping and having reread the architecture and pitfalls mentioned in code, I am running it with 2 layers in critic and generator, no batch norm in generator and batch norm plus dropout in critic. I have attached code as shown below:
self.dense = tf.keras.layers.Dense(512 * 32 * 32, use_bias=False) self.reshape = tf.keras.layers.Reshape((32, 32, 512)) self.convT_1 = tf.keras.layers.Conv2DTranspose(256, (4,4), strides=(2, 2), padding='same', use_bias=False) self.convT_2 = tf.keras.layers.Conv2DTranspose(128, (4,4), strides=(2, 2), padding='same', use_bias=False) self.convT_6 = tf.keras.layers.Conv2D(self.channels, (3, 3), padding='same', use_bias=False, activation='tanh')
which I call as:
x = self.dense(inputs) #x = self.batch_norm_1(x, training) x = self.leakyrelu_1(x) x = self.reshape(x) x = self.convT_1(x) #x = self.batch_norm_2(x, training) x = self.leakyrelu_2(x) x = self.convT_2(x) #x = self.batch_norm_3(x, training) x = self.leakyrelu_3(x) return self.convT_6(x)
Now the trade off that I mention:
I am running my code on Google Collab with GPU, using a paid for GPU or a real GPU is not an option unfortunately and I am restricted.
For the current model, I observe: Terrible images, where they are nothing but blurs. But good loss function. The generator rises to some positive value, drops and rises while critic loss does the opposite, so the functionality is not compromised.
My assumption is this comes down to the number of filters, two layers worth of filters just isn't enough to generate the required data. It can work for MNIST which is simple enough, not this
The other thing I tried:
Deepening my model and changing the weight clipping(0.001 to 0.1)
I found that the images are of a usable quality but the losses do not show required trends. Here I have enough filters and a good representation/decent one is learned. But we do not have the losses as needed.
My guess: As the paper says, changing clipping affects how long a model can take to reach equilibrium and stabilise, mine would just take too long thanks to the depth. So if I had more time, I could do it. But because of Colab, I do not have said time to do it.
As a third option: I went to my original code and just added more filters to each layer. 256 became 512 and so on. I tried it and got an OOM error from colab, saying that tensors are too large to work with.
Given that I am working on a comparative study, my actual results barely matter as much as comparing. So I can always talk about these issues, but I still need to find a middle ground where I can show a mix of both. Maybe even get better images as the WGAN is getting outperformed by things like the DC-GAN and I am at a loss for words. Maybe I can just plot and get blurry images and leave it as a proof of concept and talk about short comings.
What I am trying to look into is methods to get my model training faster. Maybe something that says the exact same model with a slight change could yield much quicker results. So say, doing something like XYZ takes epochs required down and time per epoch down to get the desired results. Because right now it is a mess
Open to help about either aspects, my paper writing or my results.
Although on Data science I am expected the latter. Thanks in advance