I'm training a two layer neural network and outputting the cost function during iteration and noticed that the cost function increases dramatically with increasing iteration number. Initially I suspect it's because the learning rate of gradient descent is too high, so I changed from 0.05 to 0.005. However, this doesn't help at all. Any suggestion is highly appreciated!

Below is part of my code:

# construct model
y_pred = multilayer_perceptron(x, weights, biases)

# define cost function(mean squred error) and optimizer(gradient descent)
cost = tf.reduce_mean(tf.square(y - y_pred))
optimizer = tf.train.MomentumOptimizer(learning_rate = learning_rate, 
            momentum = momentum).minimize(cost)

EPOCHS = 100
# initialize parameters
init_op = tf.global_variables_initializer()

with tf.Session() as sess:
    for epoch_no in range(EPOCHS):
        _, c = sess.run([optimizer, cost], feed_dict={x:X_train , y:Y_train})
       print('Epoch number: {}, cost: {}'.format(epoch_no, c))

# For running test dataset
results, test_cost = sess.run([y_pred, cost], feed_dict={x:X_train, y:Y_train})
print('test cost: {:.3f}'.format(test_cost))

Output for the first 3 Epoches:

Epoch number: 0, cost: 509.89886474609375
Epoch number: 1, cost: 287486752.0
Epoch number: 2, cost: 2.262859251393233e+18

2 Answers 2


It turned out that my activation function is not well chosen. I chose a linear activation function for the output layer where indeed a ReLu should be chosen. The cost function decreases well after I changed the activation function. I guess this solution works case by case though. May it helps.


I had similar problems and for me the reason was that my regularization parameter lambda was too high. I removed regularization completely and set it to 0 (that would be a good starting point). I was working on the IRIS dataset classification, and after that i finally managed to get 100% accuracy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.