(The lower right plots just give the last 100 iterations, and the x axis is always the iteration.)
Once it gets close to the bottom, the net starts overshooting and then correcting itself -- eventually working its way to the bottom (or a bottom) of the loss surface). This is annoying -- I'd like it to go faster and not dither.
Features of the net:
- N = about 7000, about 2000 input variables, 10 layers of 100 nodes each
- The loss is L2-penalized squared error. This one is only very lightly penalized (and thus will almost certainly end up overfitting given the architecture)
- This particular net is using full-batch gradient descent, no dropout, and full-connections.
- But I see the same thing when using minibatch, and either dropout or convolutional architectures.
- The learning rate gets multiplied by a constant every time the loss goes down, and divided by the same constant squared when the loss goes up.
- Activation is the leaky ReLU
- No batch normalization
- RMSprop for adaptive learning rate
- Fit with this package (I'm the author)
I'm not sure how to correct this behavior, and I'd appreciate suggestions. It would seem like the adaptive stepsize algorithm would need to know when it is going to hit a wall in a given direction, before it hits that wall. Is that possible? Would something like adagrad improve over RMSprop? Alternatively, I haven't yet implemented batch normalization -- not sure how that would help, but, would it for some reason?