I am a beginner in machine learning. I have built a logistic classifier in Python using TensorFlow to train on notMNIST dataset. My code is as such:

weights = tf.Variable(tf.truncated_normal(shape = [784, 10]))
bias = tf.Variable(tf.zeros(shape = [10]))
logits = tf.matmul(features, weights) + bias
prediction = tf.nn.softmax(logits)
cross_entropy = -tf.reduce_sum(labels * tf.log(prediction), reduction_indices=1)
loss = tf.reduce_mean(cross_entropy)

train_feed_dict = {features: train_features, labels: train_labels}
valid_feed_dict = {features: valid_features, labels: valid_labels}
test_feed_dict = {features: test_features, labels: test_labels}

is_correct_prediction = tf.equal(tf.argmax(prediction, 1), tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(is_correct_prediction, tf.float32))

epochs = 5
batch_size = 50
learning_rate = 0.1

optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

validation_accuracy = 0.0

with tf.Session() as session:

    batch_count = int(math.ceil(len(train_features)/batch_size))

    for epoch_i in range(epochs):

        for i in range(batch_count):

            session.run(optimizer, feed_dict = train_feed_dict)
            print(session.run(accuracy, feed_dict = train_feed_dict))

However, the problem is that while the training loss is decreasing continuously, the accuracy wavers initially, and then finally stagnates (at around 0.062). I am not able to understand what's wrong with the code. Any help would be appreciated. Thanks.

  • $\begingroup$ Welcome to Data science SE! could you post a minimal reproducible example? This is quite a lot of code for someone to look at. Thanks $\endgroup$
    – Stereo
    Jan 9 '17 at 16:56
  • 2
    $\begingroup$ I am sorry, in my hurry to get a second opinion I didn't realise what a mess I had posted. I'll edit it right now $\endgroup$
    – James Bond
    Jan 9 '17 at 17:12
  • $\begingroup$ @James Bond Every thing seems alright to me, but you should try initializing bias with non-zero values and lowering the learning_rate. $\endgroup$ Feb 20 '17 at 16:11

Use tf.softmax_cross_entropy_with_logits function instead of writing it by yourself because tf.softmax_cross_entropy_with_logits is more computationally stable.

Try this, loss = tf.reduce_mean(tf.softmax_cross_entropy_with_logits(logits))

Refrence: https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.