To design a neural network based predictor for f(x) = sin(x) I designed the following network. Input ----> Dense(num_neurons=1) ---> Relu() ----> Dense(num_neurons=100) ---> Relu() ---> Dense(num_neurons=1) ----> Relu()--> Output.

The choice of loss was L2 and optimizer was standard gradient descent. The predictor was trained on X_train = numpy.arange(0.0, 314.1, 0.1) and Y_train = numpy.sin(X_train). It is subsequently tested on X_test= numpy.arange(-10.0, 10.0, 0.001) and Y_test = numpy.sin(X_test) The predictor however performs badly on the test data. What could have gone wrong?

The training data is too small to train the network. Adding more data especially when X < 0

Relu activations used in the network are wrong. Changing the activation in the output node from Relu to Linear should help network learn the function better.

Network is not deep enough. Adding more hidden units/layers to the network will help the network generalize

L2 loss is not a good metric to measure loss when regressing for sin(x)

Gradient descent is not the best optimizer for this problem. Adam optimizer would be a better option.

This type of neural network architecture can fundamentally not learn a function like sin(x).

  • 1
    $\begingroup$ Don't have a relu the last just before the output.. $\endgroup$
    – Aditya
    Feb 23 '18 at 6:45

Since we know that sin(x) can have -ve values as well, Relu will kill all those mercilessly..

Also we can't have relu just before the output layer as it doesn't makes any sense..(Atleast I have never seen this architecture before..)

Also you can make that 100 drop to around 40

Check this link also

  • $\begingroup$ The parameter out in front of a ReLU activation function can be less than zero, so sine is perfectly reasonable to approximate (universal approximation theorem). $\endgroup$
    – Dave
    Feb 8 '20 at 3:55
  • $\begingroup$ Yes Dave, it can be approximated since it has a expansion series as well but if we will have ReLU just before our preds, we won't get -ve values, isn't? $\endgroup$
    – Aditya
    Feb 8 '20 at 3:57
  • $\begingroup$ I didn’t catch that you meant the ReLU on the last neuron. That one will kill half of the number line, yes, but it could be either half (consider if the parameter out in front is 4 versus -4). $\endgroup$
    – Dave
    Feb 8 '20 at 4:09
  • $\begingroup$ Yep I agree! Thanks for passing by; Just curious, wouldn't a TanH be better? Given -1->1 range $\endgroup$
    – Aditya
    Feb 9 '20 at 2:05
  • $\begingroup$ I’m not sure that you need an activation function there. The previous layer(s) should be able to figure out the curve. Then just add them up in the output node. $\endgroup$
    – Dave
    Feb 9 '20 at 2:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.