0
$\begingroup$

I'm trying to use an MLP to approximate a smooth function f : R^3 -> R, that takes a point in space as an argument, and returns a scalar value.

The MLP architecture has a 3-dimensional (for 3 point coordinates) input layer, N hidden layers and a single linear scalar output layer, since the output should be the function value:

   x  x      x
x  x  x      x
x  x  x  ... x  x
x  x  x      x
   x  x      x

I'm using the tanh activation function because I want the model (MLP) to be continuously differential.

I'm playing with different hidden layer architectures, using the Adam solver, and I get this behavior for the MSE loss

enter image description here

The maximal validation error that I get with this mean MSE loss is 99.982942% - is this generally considered accurate for regression?

For the network with hidden layers (16, 16, 16, 16, 16), the error stagnates but then drops and oscillates. I suspect the oscillation is due to the diminishing gradients when using the tanh activation function for learning, is this true?

How to set/improve the learning rate, are there techniques that prevent oscillations when the solver (Adam, SGD, ...) approaches the optimum?

Improve this question
$\endgroup$
2
  • $\begingroup$ What exactly is your question? $\endgroup$
    – spectre
    Dec 20, 2021 at 12:19
  • $\begingroup$ @spectre: I'm interested in setting/improving the learning rate, so that I avoid the oscillations when the solver approaches the optimum. I edited the question. $\endgroup$
    – tmaric
    Dec 20, 2021 at 14:58

1 Answer 1

Reset to default
1
$\begingroup$

If your goal is to set the optimum value of hyperparameters (weather it be learning rate, no of layers, activation function etc.) you should look into Keras Tuner.

The reason as to why the learning curve is oscillating is not clear to me without seeing your code/data. But if you want to set an optimum value of learning rate then look into Kera Tuner provided you ARE using keras + tensorflow.

If you are using PyTorch as the frame work then you could use any of the following HP tuning methods/packages:

1.)Ray Tune

2.) Optuna

3.) Auto Pytorch

4.) BoTorch

There are many more packages. You could do a google search.

Cheers!

Improve this answer
$\endgroup$
2
  • $\begingroup$ I'm using Pytorch, is there something equivalent? When I reduce the learning rate of the solver, the oscillations reduce significantly (as expected).... $\endgroup$
    – tmaric
    Dec 21, 2021 at 11:25
  • $\begingroup$ Edited my answer to reflect your comment $\endgroup$
    – spectre
    Dec 21, 2021 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.