Accuracy stagnated while training notMNIST data

I am a beginner in machine learning. I have built a logistic classifier in Python using TensorFlow to train on notMNIST dataset. My code is as such:

weights = tf.Variable(tf.truncated_normal(shape = [784, 10]))
bias = tf.Variable(tf.zeros(shape = [10]))
logits = tf.matmul(features, weights) + bias
prediction = tf.nn.softmax(logits)
cross_entropy = -tf.reduce_sum(labels * tf.log(prediction), reduction_indices=1)
loss = tf.reduce_mean(cross_entropy)

train_feed_dict = {features: train_features, labels: train_labels}
valid_feed_dict = {features: valid_features, labels: valid_labels}
test_feed_dict = {features: test_features, labels: test_labels}

is_correct_prediction = tf.equal(tf.argmax(prediction, 1), tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(is_correct_prediction, tf.float32))

epochs = 5
batch_size = 50
learning_rate = 0.1

validation_accuracy = 0.0

with tf.Session() as session:

session.run(tf.global_variables_initializer())
batch_count = int(math.ceil(len(train_features)/batch_size))

for epoch_i in range(epochs):

for i in range(batch_count):

session.run(optimizer, feed_dict = train_feed_dict)
print(session.run(accuracy, feed_dict = train_feed_dict))


However, the problem is that while the training loss is decreasing continuously, the accuracy wavers initially, and then finally stagnates (at around 0.062). I am not able to understand what's wrong with the code. Any help would be appreciated. Thanks.

• Welcome to Data science SE! could you post a minimal reproducible example? This is quite a lot of code for someone to look at. Thanks Jan 9 '17 at 16:56
• I am sorry, in my hurry to get a second opinion I didn't realise what a mess I had posted. I'll edit it right now Jan 9 '17 at 17:12
• @James Bond Every thing seems alright to me, but you should try initializing bias with non-zero values and lowering the learning_rate. Feb 20 '17 at 16:11

Use tf.softmax_cross_entropy_with_logits function instead of writing it by yourself because tf.softmax_cross_entropy_with_logits is more computationally stable.
Try this, loss = tf.reduce_mean(tf.softmax_cross_entropy_with_logits(logits))