I am a beginner in machine learning. I have built a logistic classifier in Python using TensorFlow to train on notMNIST dataset. My code is as such:

weights = tf.Variable(tf.truncated_normal(shape = [784, 10]))
bias = tf.Variable(tf.zeros(shape = [10]))
logits = tf.matmul(features, weights) + bias
prediction = tf.nn.softmax(logits)
cross_entropy = -tf.reduce_sum(labels * tf.log(prediction), reduction_indices=1)
loss = tf.reduce_mean(cross_entropy)

train_feed_dict = {features: train_features, labels: train_labels}
valid_feed_dict = {features: valid_features, labels: valid_labels}
test_feed_dict = {features: test_features, labels: test_labels}

is_correct_prediction = tf.equal(tf.argmax(prediction, 1), tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(is_correct_prediction, tf.float32))

epochs = 5
batch_size = 50
learning_rate = 0.1

optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

validation_accuracy = 0.0

with tf.Session() as session:

    batch_count = int(math.ceil(len(train_features)/batch_size))

    for epoch_i in range(epochs):

        for i in range(batch_count):

            session.run(optimizer, feed_dict = train_feed_dict)
            print(session.run(accuracy, feed_dict = train_feed_dict))

However, the problem is that while the training loss is decreasing continuously, the accuracy wavers initially, and then finally stagnates (at around 0.062). I am not able to understand what's wrong with the code. Any help would be appreciated. Thanks.

  • $\begingroup$ Welcome to Data science SE! could you post a minimal reproducible example? This is quite a lot of code for someone to look at. Thanks $\endgroup$
    – Stereo
    Commented Jan 9, 2017 at 16:56
  • 2
    $\begingroup$ I am sorry, in my hurry to get a second opinion I didn't realise what a mess I had posted. I'll edit it right now $\endgroup$
    – James Bond
    Commented Jan 9, 2017 at 17:12
  • $\begingroup$ @James Bond Every thing seems alright to me, but you should try initializing bias with non-zero values and lowering the learning_rate. $\endgroup$ Commented Feb 20, 2017 at 16:11

1 Answer 1


Use tf.softmax_cross_entropy_with_logits function instead of writing it by yourself because tf.softmax_cross_entropy_with_logits is more computationally stable.

Try this, loss = tf.reduce_mean(tf.softmax_cross_entropy_with_logits(logits))

Refrence: https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.