In Introduction I have just changed
loss = tf.reduce_mean(tf.square(y - y_data))
loss = tf.reduce_mean(tf.abs(y - y_data))
and model is unable to learn the loss just became bigger with time. Why?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
I tried this and got same result.
It is because the gradient of
.abs is harder for a simple optimiser to follow to the minima, unlike squared difference where gradient approaches zero slowly, the gradient of the absolute difference has a fixed magnitude which abruptly reverses, which tends to make the optimiser oscillate around the minimum point. Basic gradient descent is very sensitive to magnitude of the gradient, and to the learning rate, which is essentially just a multiplier of the gradient for step sizes.
The simplest fix is to reduce the learning rate e.g. change line
optimizer = tf.train.GradientDescentOptimizer(0.5)
optimizer = tf.train.GradientDescentOptimizer(0.05)
Also, have a play with different optimisers. Some will be able to cope with
.abs-based loss better.