I'm trying to train a GAN, and the architecture includes a fully connected layer before the output activation function. In my case, by the second training iteration this layer's output always explodes.
I have not been able to identify any causes through debugging. The layer implementation is extremely simple:
def fully_connected_d(inp, output_dim, name="Linear", stddev=0.02):
shape = inp.get_shape().as_list()
with tf.variable_scope(name):
w = tf.get_variable("w", [shape[1], output_dim], initializer=tf.truncated_normal_initializer(stddev=stddev))
#biases = tf.get_variable("b", [output_dim], initializer=tf.constant_initializer(0.0))
return tf.matmul(inp, w) #tf.nn.bias_add(tf.matmul(inp, w), biases)
Here I've removed bias to rule out bias explosion as a cause, but this occurs with or without bias.
In any given run, no matter what the input is to this layer, its activation jumps from around zero to roughly around 10 within the first few training iterations:
I can't figure out why the output is in this range, as the weights and the inputs are fairly tiny:
Weights:
Inputs:
Finally, I checked the gradients of loss with respect to the weights, and they also appear to be sane before activations jump. Once activations are increased they go towards zero because I'm using $\tanh$ and its gradient is small for extremely large inputs.
Essentially, I'm unsure how (seemingly sane input) x (seemingly sane weights) yields (insane activations). Because the output jumps so quickly I think it would be a design flaw, but the layer itself is so simple that I can't see what I'm missing. Any help is appreciated.
