2
$\begingroup$

I'm learning about variational autoencoders and I've implemented a simple example in keras, model summary below. I've copied the loss function from one of Francois Chollet's blog posts and I'm getting really really negative losses. What am I missing here?

    Model: "model_1"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to
==================================================================================================
input_1 (InputLayer)            [(None, 224)]        0
__________________________________________________________________________________________________
encoding_flatten (Flatten)      (None, 224)          0           input_1[0][0]
__________________________________________________________________________________________________
encoding_layer_2 (Dense)        (None, 256)          57600       encoding_flatten[0][0]
__________________________________________________________________________________________________
encoding_layer_3 (Dense)        (None, 128)          32896       encoding_layer_2[0][0]
__________________________________________________________________________________________________
encoding_layer_4 (Dense)        (None, 64)           8256        encoding_layer_3[0][0]
__________________________________________________________________________________________________
encoding_layer_5 (Dense)        (None, 32)           2080        encoding_layer_4[0][0]
__________________________________________________________________________________________________
encoding_layer_6 (Dense)        (None, 16)           528         encoding_layer_5[0][0]
__________________________________________________________________________________________________
encoder_mean (Dense)            (None, 16)           272         encoding_layer_6[0][0]
__________________________________________________________________________________________________
encoder_sigma (Dense)           (None, 16)           272         encoding_layer_6[0][0]
__________________________________________________________________________________________________
lambda (Lambda)                 (None, 16)           0           encoder_mean[0][0]
                                                                 encoder_sigma[0][0]
__________________________________________________________________________________________________
decoder_layer_1 (Dense)         (None, 16)           272         lambda[0][0]
__________________________________________________________________________________________________
decoder_layer_2 (Dense)         (None, 32)           544         decoder_layer_1[0][0]
__________________________________________________________________________________________________
decoder_layer_3 (Dense)         (None, 64)           2112        decoder_layer_2[0][0]
__________________________________________________________________________________________________
decoder_layer_4 (Dense)         (None, 128)          8320        decoder_layer_3[0][0]
__________________________________________________________________________________________________
decoder_layer_5 (Dense)         (None, 256)          33024       decoder_layer_4[0][0]
__________________________________________________________________________________________________
decoder_mean (Dense)            (None, 224)          57568       decoder_layer_5[0][0]
==================================================================================================
Total params: 203,744
Trainable params: 203,744
Non-trainable params: 0
__________________________________________________________________________________________________
Train on 3974 samples, validate on 994 samples
Epoch 1/10
3974/3974 [==============================] - 3s 677us/sample - loss: -28.1519 - val_loss: -33.5864
Epoch 2/10
3974/3974 [==============================] - 1s 346us/sample - loss: -137258.8175 - val_loss: -3683802.1489
Epoch 3/10
3974/3974 [==============================] - 1s 344us/sample - loss: -14543022903.6056 - val_loss: -107811177469.9396
Epoch 4/10
3974/3974 [==============================] - 1s 363us/sample - loss: -3011718676570.7012 - val_loss: -13131454938476.6816
Epoch 5/10
3974/3974 [==============================] - 1s 350us/sample - loss: -101442605943572.4844 - val_loss: -322685056398605.9375
Epoch 6/10
3974/3974 [==============================] - 1s 344us/sample - loss: -1417424385529640.5000 - val_loss: -3687688508198145.5000
Epoch 7/10
3974/3974 [==============================] - 1s 358us/sample - loss: -11794297368126698.0000 - val_loss: -26632844827070784.0000
Epoch 8/10
3974/3974 [==============================] - 1s 339us/sample - loss: -69508229806130784.0000 - val_loss: -141312065640756336.0000
Epoch 9/10
3974/3974 [==============================] - 1s 345us/sample - loss: -319838384005810432.0000 - val_loss: -599553350073361152.0000
Epoch 10/10
3974/3974 [==============================] - 1s 342us/sample - loss: -1221653451351326464.0000 - val_loss: -2147128507956525312.0000

latent sample func:

def sampling(self,args):
    """Reparameterization trick by sampling fr an isotropic unit Gaussian.
    # Arguments
        args (tensor): mean and log of variance of Q(z|X)
    # Returns
        z (tensor): sampled latent vector
    """

    z_mean, z_log_var = args
    set = tf.shape(z_mean)[0]
    batch = tf.shape(z_mean)[1]
    dim = tf.shape(z_mean)[-1]
    # by default, random_normal has mean=0 and std=1.0
    epsilon = tf.random.normal(shape=(set, dim))#tfp.distributions.Normal(mean=tf.zeros(shape=(batch, dim)),loc=tf.ones(shape=(batch, dim)))
    return z_mean + (z_log_var * epsilon)

Loss func:

def vae_loss(self,input, x_decoded_mean):
    xent_loss = tf.reduce_mean(tf.keras.backend.binary_crossentropy(input, x_decoded_mean))
    kl_loss = -0.5 * tf.reduce_sum(tf.square(self.encoded_mean) + tf.square(self.encoded_sigma) - tf.math.log(tf.square(self.encoded_sigma)) - 1, -1)
    return xent_loss + kl_loss

Another vae_loss implementation:

def vae_loss(self,input, x_decoded_mean):
    gen_loss = tf.reduce_sum(tf.keras.backend.binary_crossentropy(input, x_decoded_mean))
    #gen_loss = tf.losses.mean_squared_error(input,x_decoded_mean)
    kl_loss = -0.5 * tf.reduce_sum(1 + self.encoded_sigma - tf.square(self.encoded_mean) - tf.exp(self.encoded_sigma), -1)
    return tf.reduce_mean(gen_loss + kl_loss)

log_sigma kl_loss:

kl_loss = 0.5 * tf.reduce_sum(tf.square(self.encoded_mean) + tf.square(tf.exp(self.encoded_sigma)) - self.encoded_sigma - 1, axis=-1)
$\endgroup$
7
  • 2
    $\begingroup$ Welcome to StackExchange.DS! KL loss must be minimized, I think it should be +0.5 to decrease the mean and std toward 0 and 1 respectively. Let me know if this was the problem $\endgroup$
    – Esmailian
    Apr 5 '19 at 16:43
  • $\begingroup$ @Esmailian Thanks for the suggestion. Unfortunately, no that doesn't help. I think there is a problem with my implementation of KL divergence and/or the sampling. There are a lot of implementations where both or one of the loss components is negative: blog.keras.io/building-autoencoders-in-keras.html, jmetzen.github.io/2015-11-27/vae.html, etc. Not really sure where the difference lies, but I'm expecting that this is due to slight variations in the overall implementation $\endgroup$
    – Jed
    Apr 5 '19 at 18:15
  • 2
    $\begingroup$ In Keras by Francois Chollet, the terms inside K.mean are negative of yours, that's why -0.5 works for them. $\endgroup$
    – Esmailian
    Apr 5 '19 at 18:18
  • 2
    $\begingroup$ Also, another trick is that we let the network produce log(sigma) instead of sigma and then exponentiate it (the same as what Francois Chollet does) for stability. Take a look at side notes of this answer. $\endgroup$
    – Esmailian
    Apr 5 '19 at 18:24
  • $\begingroup$ thanks, if you make the network generate log_sigma, then is you loss function going to work out to the above log_sigma kl_loss? $\endgroup$
    – Jed
    Apr 5 '19 at 18:43
0
$\begingroup$

Is your data is binary data. I think the binary_crossentropy loss is suited for the binary input. All the data in MNIST is binary. If your input is continuous like color image, try MSE loss or others. check it here https://github.com/Lasagne/Recipes/issues/54

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.