I'm trying to use an MLP to approximate a smooth function f : R^3 -> R, that takes a point in space as an argument, and returns a scalar value.
The MLP architecture has a 3-dimensional (for 3 point coordinates) input layer, N hidden layers and a single linear scalar output layer, since the output should be the function value:
x x x x x x x x x x ... x x x x x x x x x
I'm using the tanh activation function because I want the model (MLP) to be continuously differential.
I'm playing with different hidden layer architectures, using the Adam solver, and I get this behavior for the MSE loss
The maximal validation error that I get with this mean MSE loss is 99.982942% - is this generally considered accurate for regression?
For the network with hidden layers (16, 16, 16, 16, 16), the error stagnates but then drops and oscillates. I suspect the oscillation is due to the diminishing gradients when using the tanh activation function for learning, is this true?
How to set/improve the learning rate, are there techniques that prevent oscillations when the solver (Adam, SGD, ...) approaches the optimum?