1
$\begingroup$

I'm training a two layer neural network and outputting the cost function during iteration and noticed that the cost function increases dramatically with increasing iteration number. Initially I suspect it's because the learning rate of gradient descent is too high, so I changed from 0.05 to 0.005. However, this doesn't help at all. Any suggestion is highly appreciated!

Below is part of my code:

# construct model
y_pred = multilayer_perceptron(x, weights, biases)

# define cost function(mean squred error) and optimizer(gradient descent)
cost = tf.reduce_mean(tf.square(y - y_pred))
optimizer = tf.train.MomentumOptimizer(learning_rate = learning_rate, 
            momentum = momentum).minimize(cost)

EPOCHS = 100
# initialize parameters
init_op = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init_op)
    for epoch_no in range(EPOCHS):
        _, c = sess.run([optimizer, cost], feed_dict={x:X_train , y:Y_train})
       print('Epoch number: {}, cost: {}'.format(epoch_no, c))

# For running test dataset
results, test_cost = sess.run([y_pred, cost], feed_dict={x:X_train, y:Y_train})
print('test cost: {:.3f}'.format(test_cost))
print(y_pred)

Output for the first 3 Epoches:

Epoch number: 0, cost: 509.89886474609375
Epoch number: 1, cost: 287486752.0
Epoch number: 2, cost: 2.262859251393233e+18
$\endgroup$
1
$\begingroup$

It turned out that my activation function is not well chosen. I chose a linear activation function for the output layer where indeed a ReLu should be chosen. The cost function decreases well after I changed the activation function. I guess this solution works case by case though. May it helps.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.