1
$\begingroup$

The code here:

https://github.com/keras-team/keras/blob/master/examples/variational_autoencoder.py

Specifically line 53:

xent_loss = original_dim * metrics.binary_crossentropy(x, x_decoded_mean)

Why is the cross entropy multiplied by original_dim? Also, does this function calculate cross entropy only across the batch dimension (I noticed there is no axis input)? It's hard to tell from the documentation...

$\endgroup$

2 Answers 2

1
$\begingroup$

Keras metrics.binary_crossentropy computes the cross entropy averaged across all inputs (pseudocode):

original_dim = 3 
x = [1,1,0]
x_decoded = [0.2393,0.7484,-1.1399]

average_BCE = binary_crossentropy(x, x_decoded)
print(average_BCE)
>>>0.1186

For this part of autoencoder loss we need the sum, not the average over all squared differences between input and output pixels, which is equivalent to average_crossentropy_of_pixels * num_pixels (original_dim)

print(original_dim * average_BCE)
>>>0.3559

Another way of writing this part would (which I think is more illustrative of what's happening, but probably less performant in Keras land):

xent_loss = K.sum(K.square(x - sigmoid(x_decoded)))
print(xent_loss)
>>>0.3559

Regarding the second part, since the first operation you are really doing is subtraction, it implied the tensors for input and output are the same size. This can be found if you check the implementation code - which is better than the docs in this case - go to line 3056: https://github.com/keras-team/keras/blob/master/keras/backend/tensorflow_backend.py

$\endgroup$
5
  • $\begingroup$ If I actually wanted to evaluate what the output of metrics.binary_crossentropy with x = [1,1,0] and x_decoded = [0.2393,0.7484,-1.1399]. How would I write that in keras? Sorry if its a nubile question but I can't seem to find any simple guides on it. This would help me test out functions and see the outputs and be able to debug stuff like this better... $\endgroup$ Commented Apr 24, 2018 at 15:54
  • $\begingroup$ No Idea LOL. I stopped using keras for that exact reason ... I use pytorch, which allows dynamic gpu code compilation unlike K and TF. There is a way to do it in keras which is straight forward, but this is a separate Q. I would try a separate file with just those inputs into a model with one layer which is initialized to all one's? Anyone? $\endgroup$ Commented Apr 24, 2018 at 16:09
  • 1
    $\begingroup$ Ok I ended up getting it. I'll post a complementary answer. Your example is a smidge confusing since the binary cross entropy performs clipping on x_decoded to make sure values are positive and between 0 and 1. $\endgroup$ Commented Apr 24, 2018 at 16:12
  • $\begingroup$ I'm accepting your answer because its correct. I've supplemented with my answer so people can test it out. I'll also look into pytorch... $\endgroup$ Commented Apr 24, 2018 at 16:22
  • $\begingroup$ Yup you are right. Sorry about that - slightly bad example on my part, should have realized that the values being predicted are between 0 and 1 as they are pixels in the end. the negative just adds confusion, $\endgroup$ Commented Apr 24, 2018 at 16:42
0
$\begingroup$
# Do Keras' binary cross entropy
x = Input(shape=(3,))
x_decoded = Input(shape=(3,))  
bce = metrics.binary_crossentropy(x,x_decoded)
sess = K.get_session()
with sess.as_default():
    print(bce.eval(feed_dict={x: np.array([[1,1,0]]),
                              x_decoded: np.array([[0.2393,0.7484,-1.1399]])}))

# Do the same thing in numpy directly
epsilon = 1e-7
x = np.array([1,1,0]) 
x_decoded = np.array([0.2393,0.7484,-1.1399])

output = np.clip(x_decoded, epsilon, 1-epsilon)
output = -x * np.log(output) - (1 - x) * np.log(1 - output)
print(np.mean(output, axis=-1))

Output:

[0.57328504]
0.5732850228833389

So the problem is the mean. It should be a sum. And also I guess they do some stuff with epsilon to prevent weird edge cases that can make it explode.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.