Why does cost function on a neural network increase?

Question

I'm training a two layer neural network and outputting the cost function during iteration and noticed that the cost function increases dramatically with increasing iteration number. Initially I suspect it's because the learning rate of gradient descent is too high, so I changed from 0.05 to 0.005. However, this doesn't help at all. Any suggestion is highly appreciated!

Below is part of my code:

# construct model
y_pred = multilayer_perceptron(x, weights, biases)

# define cost function(mean squred error) and optimizer(gradient descent)
cost = tf.reduce_mean(tf.square(y - y_pred))
optimizer = tf.train.MomentumOptimizer(learning_rate = learning_rate, 
            momentum = momentum).minimize(cost)

EPOCHS = 100
# initialize parameters
init_op = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init_op)
    for epoch_no in range(EPOCHS):
        _, c = sess.run([optimizer, cost], feed_dict={x:X_train , y:Y_train})
       print('Epoch number: {}, cost: {}'.format(epoch_no, c))

# For running test dataset
results, test_cost = sess.run([y_pred, cost], feed_dict={x:X_train, y:Y_train})
print('test cost: {:.3f}'.format(test_cost))
print(y_pred)

Output for the first 3 Epoches:

Epoch number: 0, cost: 509.89886474609375
Epoch number: 1, cost: 287486752.0
Epoch number: 2, cost: 2.262859251393233e+18

Answer 1

It turned out that my activation function is not well chosen. I chose a linear activation function for the output layer where indeed a ReLu should be chosen. The cost function decreases well after I changed the activation function. I guess this solution works case by case though. May it helps.

Answer 2

I had similar problems and for me the reason was that my regularization parameter lambda was too high. I removed regularization completely and set it to 0 (that would be a good starting point). I was working on the IRIS dataset classification, and after that i finally managed to get 100% accuracy.