# Strange behavior with Adam optimizer when training for too long

I'm trying to train a single perceptron (1000 input units, 1 output, no hidden layers) on 64 randomly generated data points. I'm using Pytorch using the Adam optimizer:

import torch

torch.manual_seed(545345)
N, D_in, D_out = 64, 1000, 1

x = Variable(torch.randn(N, D_in))
y = Variable(torch.randn(N, D_out))

model = torch.nn.Linear(D_in, D_out)
loss_fn = torch.nn.MSELoss(size_average=False)

for t in xrange(5000):
y_pred = model(x)
loss = loss_fn(y_pred, y)

print(t, loss.data[0])

loss.backward()
optimizer.step()


Initially, the loss quickly decreases, as expected:

(0, 91.74887084960938)
(1, 76.85824584960938)
(2, 63.434078216552734)
(3, 51.46927261352539)
(4, 40.942893981933594)
(5, 31.819372177124023)


Around 300 iterations, the error reaches near zero:

(300, 2.1734419819452455e-12)
(301, 1.90354676465887e-12)
(302, 2.3347573874232808e-12)


This goes on for a few thousand iterations. However, after training for too long, the error starts to increase again:

(4997, 0.002102422062307596)
(4998, 0.0020302983466535807)
(4999, 0.0017039275262504816)


Why is this happening?

• I don't think overfitting explains it -- the training loss is going up, not the validation loss. For example, this doesn't happen when using SGD, only with Adam. – Bai Li Nov 22 '17 at 21:14
• The model has 1000 parameters and there is only 1 data point, so the model should fit the data exactly and the loss should be zero. – Bai Li Nov 22 '17 at 21:20
• Oh sorry, you're right. There are 64 data points. – Bai Li Nov 22 '17 at 21:22
• There are 64 data points (ie, constraints) and 1000 parameters, so it is possible to find choices for the parameters so that the error is zero (and this is easy to do analytically). My question is why does Adam not find this. – Bai Li Nov 22 '17 at 21:33
• – Bai Li Nov 22 '17 at 21:33

This small instability at the end of convergence is a feature of Adam (and RMSProp) due to how it estimates mean gradient magnitudes over recent steps and divides by them.

One thing Adam does is maintain a rolling geometric mean of recent gradients and squares of the gradients. The squares of the gradients are used to divide (another rolling mean of) the current gradient to decide the current step. However, when your gradient becomes and stays very close to zero, this will make the squares of the gradient become so low that they either have large rounding errors or are effectively zero, which can introduce instability (for instance a long-term stable gradient in one dimension makes a relatively small step from $10^{-10}$ to $10^{-5}$ due to changes in other params), and the step size will start to jump around, before settling again.

This actually makes Adam less stable and worse for your problem than more basic gradient descent, assuming you want to get as numerically close to zero loss as calculations allow for your problem.

In practice on deep learning problems, you don't get this close to convergence (and for some regularisation techniques such as early stopping, you don't want to anyway), so it is usually not a practical concern on the types of problem that Adam was designed for.

You can actually see this occurring for RMSProp in a comparison of different optimisers (RMSProp is the black line - watch the very last steps just as it reaches the target):

You can make Adam more stable and able to get closer to true convergence by reducing the learning rate. E.g.

optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)


It will take longer to optimise. Using lr=1e-5 you need to train for 20,000+ iterations before you see the instability and the instability is less dramatic, values hover around $10^{-7}$.

• This is a spectacular visualization, Neil. What are the actual dimensions? What is x and y representing? Are the frames some delta t or n epochs per frame? I'm guessing the star is the global optimum in a topographical representation of disparity (error) in relation to two selected parameters. Is my guess correct? – Douglas Daseeco Sep 28 '18 at 9:25
• It's not my visualisation, you will find it in many places. The dimensions are arbitrary units of input parameters to a test function, and the graph shows contour lines for that function (again in arbitrary units, presumably scaled so that NN works OK). Each frame is a weight update step. It is probably equivalent to a mini-batch update, and due to behaviour of SGD, I expect that it is in fact being solved exactly using the true gradient of the test function - i.e. there is no data set or sampling. – Neil Slater Sep 28 '18 at 9:32

The reason is exactly as mentioned in the other answer with a great suggestion to use a smaller learning rate to avoid this problem around small gradients.

I can think of a couple of approaches:

1. You can clip the gradients with an upper/lower bound but this does not guarantee convergence and may result in training freeze by getting trapped in some local minima and never get out of it.

2. Train with a higher batch size, more epochs and with a decayed learning rate. Now I do not have any practical proof that increasing a batch size results in better gradients but from what I had observed by facing problems similar to yours, doing so has almost always helped.

I am sure there are other methods (like cyclical learning rate etc) that try to find an optimal learning rate based on statistics.