# Implement MLP in tensorflow

There are many resources online about how to implement MLP in tensorflow, and most of the samples do work :) But I am interested in a particular one, that I learned from https://www.coursera.org/learn/machine-learning. In which, it uses a cost function defined as follow:

$J(\theta) = \frac{1}{m} \sum_{i=1}^{m} \sum_{k=1}^{K} \left[ -y_k^{(i)} \log((h_\theta(x^{(i)}))_k - (1 - y_k^{(i)}) \log(1 - (h_\theta(x^{(i)}))_k \right]$

$h_\theta$ is the sigmoid function.

And there's my implementation:

# one hidden layer MLP

x = tf.placeholder(tf.float32, shape=[None, 784])
y = tf.placeholder(tf.float32, shape=[None, 10])

W_h1 = tf.Variable(tf.random_normal([784, 512]))
h1 = tf.nn.sigmoid(tf.matmul(x, W_h1))

W_out = tf.Variable(tf.random_normal([512, 10]))
y_ = tf.matmul(h1, W_out)

# cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(y_, y)
cross_entropy = tf.reduce_sum(- y * tf.log(y_) - (1 - y) * tf.log(1 - y_), 1)
loss = tf.reduce_mean(cross_entropy)

correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# train
with tf.Session() as s:
s.run(tf.initialize_all_variables())

for i in range(10000):
batch_x, batch_y = mnist.train.next_batch(100)
s.run(train_step, feed_dict={x: batch_x, y: batch_y})

if i % 100 == 0:
train_accuracy = accuracy.eval(feed_dict={x: batch_x, y: batch_y})
print('step {0}, training accuracy {1}'.format(i, train_accuracy))


I think the definition for the layers are correct, but the problem is in the cross_entropy. If I use the first one, the one got commented out, the model converges quickly; but if I use the 2nd one, which I think/hope is the translation of the previous equation, the model won't converge.

• There are several mistakes stemming from your application of the binary logistic regression model to the multinomial case (remember that MNIST has ten classes). Please follow the tensorflow MNIST guide for beginners. – Emre Jan 29 '16 at 5:06
• @Emre, it would be very appreciate if you can point out where I got wrong. Thanks~ – David S. Jan 29 '16 at 6:17
• Your loss function and its arguments are both wrong. I don't think it would work as is even if it were binary; the argument of the loss function is not normalized. Please follow the links. – Emre Jan 29 '16 at 6:28
• I implemented the model using numpy and scipy and it works. When I try to print the y_ during the training loop, the elements are all nan. I think there's some arithmetic error happened in tensorflow, but I could not figure out how to fix it. – David S. Feb 2 '16 at 13:32
• I don't have tensorflow to try it, but I think you forgot to apply the sigmoid function on y_. Try y_ = tf.nn.sigmoid(tf.matmul(h1, W_out)) instead of y_ = tf.matmul(h1, W_out). – stmax Mar 3 '16 at 7:52

1. You omitted the offset terms before the nonlinear transformations (variables b_1 and b_out). This increases the representative power of the neural network.
2. You omitted the softmax transformation at the top layer. This makes the output a probability distributions, so you can calculate the cross-entropy, which is the usual cost function for classification.
3. You used the binary form of the cross-entropy when you should have used the multi-class form.

When I run this I get accuracies over 90%:

import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

x = tf.placeholder(tf.float32, shape=[None, 784])
y = tf.placeholder(tf.float32, shape=[None, 10])

W_h1 = tf.Variable(tf.random_normal([784, 512]))
b_1 = tf.Variable(tf.random_normal())
h1 = tf.nn.sigmoid(tf.matmul(x, W_h1) + b_1)

W_out = tf.Variable(tf.random_normal([512, 10]))
b_out = tf.Variable(tf.random_normal())
y_ = tf.nn.softmax(tf.matmul(h1, W_out) + b_out)

# cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(y_, y)
cross_entropy = tf.reduce_sum(- y * tf.log(y_), 1)
loss = tf.reduce_mean(cross_entropy)

correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# train
with tf.Session() as s:
s.run(tf.initialize_all_variables())

for i in range(10000):
batch_x, batch_y = mnist.train.next_batch(100)
s.run(train_step, feed_dict={x: batch_x, y: batch_y})

if i % 1000 == 0:
train_accuracy = accuracy.eval(feed_dict={x: batch_x, y: batch_y})
print('step {0}, training accuracy {1}'.format(i, train_accuracy))

• For sure your implementation is correct. But I wanted to follow the idea from the course strictly. I omitted bias terms for I am lazy :) I think the steps to compute softmax and cross entropy is where I did wrong. – David S. Mar 3 '16 at 3:33
• You are right; the bias is "optional", but the softmax is essential. – Emre Mar 3 '16 at 3:37
• A bit late, but I believe sigmoid_cross_entropy_with_logits takes the logits (that is the previous layer without any activation), not the softmax as an input. So: tf.nn. sigmoid_cross_entropy_with_logits(tf.matmul(h1, W_out) + b_out) would be better. For single label classification (such as NIST written digits) one would use softmax_cross_entropy_with_logits. Would love to hear if I got it wrong. – Bastiaan Oct 26 '17 at 15:17