2
$\begingroup$
EPOCH=40
batch_size=50
mnist=input_data.read_data_sets("MNIST_data", one_hot=True)

tf.reset_default_graph()

input_X = tf.placeholder(tf.float32, shape=[None, 784])
input_y = tf.placeholder(tf.int64, shape=[None, 10])

input_layer=tf.layers.dense(input_X, 784, activation=tf.nn.sigmoid)
hidden1=tf.layers.dense(input_layer, 256, activation=tf.nn.sigmoid)
hidden2=tf.layers.dense(hidden1, 256, activation=tf.nn.sigmoid)
output=tf.layers.dense(hidden2, units=10)
output=tf.nn.softmax(output)


entropy=tf.nn.softmax_cross_entropy_with_logits(labels=input_y, logits=output)
loss=tf.reduce_mean(entropy)
step=tf.train.GradientDescentOptimizer(0.01).minimize(loss)

#correct=tf.nn.in_top_k(tf.argmax(output, y, 1) rank error

correct=tf.equal(tf.argmax(output, 1), tf.argmax(input_y, 1))
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))


sess=tf.InteractiveSession()
sess.run(tf.global_variables_initializer())

for epoch in range(EPOCH):
  for i in range(1000):
    X, y=mnist.train.next_batch(batch_size)
    sess.run(loss, feed_dict={input_X:X, input_y:y})
  acc_train=accuracy.eval(feed_dict={input_X:X, input_y:y})
  acc_val=accuracy.eval(feed_dict={input_X: mnist.validation.images
                                  , input_y: mnist.validation.labels})
  print(epoch, "Train accuracy: ", acc_train, "\n Val accuracy: ", acc_val)

Results went something like this-

0 Train accuracy:  0.1 
 Val accuracy:  0.0986
1 Train accuracy:  0.14 
 Val accuracy:  0.0986
2 Train accuracy:  0.06 
 Val accuracy:  0.0986
3 Train accuracy:  0.06 
 Val accuracy:  0.0986

I have referenced part of the code from O'Reily textbook on Tensorflow.

EDIT- found this O' Reily guide online-https://github.com/ageron/handson-ml/blob/master/10_introduction_to_artificial_neural_networks.ipynb

They have a code similar to mine but their's works just fine.

$\endgroup$
11
  • $\begingroup$ You have to train it much more than this. $\endgroup$ Commented Sep 8, 2018 at 5:39
  • $\begingroup$ I just posted the first few lines but my val accuracy stays the same throughout and my train accuracy never goes above 11%. I have also tried increasing the range but it only pushed the result to 10.5%. $\endgroup$ Commented Sep 8, 2018 at 5:51
  • $\begingroup$ Just change activation to relu and see the magic. $\endgroup$
    – DuttaA
    Commented Sep 8, 2018 at 8:04
  • $\begingroup$ add comments to your code. it helps people to sooner understand your code and help you. $\endgroup$
    – parvij
    Commented Sep 8, 2018 at 10:31
  • 1
    $\begingroup$ Check the input data. Is it scaled 0..255, or scaled 0..1? $\endgroup$ Commented Sep 8, 2018 at 18:17

2 Answers 2

4
$\begingroup$

Reason is you did not train the network.

You need to run the step op in your code. You defined it, but didn't use it anywhere later. Running the loss only calculates the loss, but does not train your network.

after fixing your code (remove duplicate softmax, using adadelta, run step) you should get: enter image description here

$\endgroup$
4
  • $\begingroup$ I thought this was it but for some reason even this isn't working. Nice catch though. $\endgroup$ Commented Sep 9, 2018 at 0:25
  • $\begingroup$ If you fixed that, you should be able to get >90% accuracy on both training and validation within 10 epochs, using <code>step=tf.train.AdadeltaOptimizer(0.1).minimize(loss)</code>. $\endgroup$
    – user12075
    Commented Sep 9, 2018 at 1:02
  • $\begingroup$ I got 87%. Why did the performance improve with AdadeltaOptimizer? I am glad that it did but the O' Reily code gets 97% using code similar to mine( including using GradientDescentOptimizer). Great suggestion though. $\endgroup$ Commented Sep 9, 2018 at 1:24
  • $\begingroup$ The OReily code use relu for activation instead of sigmoid. If you do that you can also get >97% within 10 epochs. $\endgroup$
    – user12075
    Commented Sep 9, 2018 at 1:43
2
$\begingroup$

I have two suggestions.

  1. Change your optimiser to AdamOptimizer.
  2. Change the number of neurons in the hidden layers. They are too many for this task. Two layers with 25 neurons for each layer will suffice.
    Omit output=tf.nn.softmax(output). Due to calculating that in the softmax_cross_entropy_with_logits. As you can read in the document of the function softmax_cross_entropy_with_logits:

WARNING: This op expects unscaled logits, since it performs a softmax on logits internally for efficiency. Do not call this op with the output of softmax, as it will produce incorrect results.

$\endgroup$
4
  • $\begingroup$ Tried that. Didn't help much. $\endgroup$ Commented Sep 8, 2018 at 18:11
  • $\begingroup$ What is your accuracy now? $\endgroup$ Commented Sep 8, 2018 at 18:48
  • $\begingroup$ It is almost 15% now $\endgroup$ Commented Sep 8, 2018 at 19:36
  • $\begingroup$ And how many epoch or iteration does it take? Also take a look at the update. $\endgroup$ Commented Sep 8, 2018 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.