To design a neural network based predictor for f(x) = sin(x) I designed the following network. Input ----> Dense(num_neurons=1) ---> Relu() ----> Dense(num_neurons=100) ---> Relu() ---> Dense(num_neurons=1) ----> Relu()--> Output.

The choice of loss was L2 and optimizer was standard gradient descent. The predictor was trained on X_train = numpy.arange(0.0, 314.1, 0.1) and Y_train = numpy.sin(X_train). It is subsequently tested on X_test= numpy.arange(-10.0, 10.0, 0.001) and Y_test = numpy.sin(X_test) The predictor however performs badly on the test data. What could have gone wrong?

The training data is too small to train the network. Adding more data especially when X < 0

Relu activations used in the network are wrong. Changing the activation in the output node from Relu to Linear should help network learn the function better.

Network is not deep enough. Adding more hidden units/layers to the network will help the network generalize

L2 loss is not a good metric to measure loss when regressing for sin(x)

Gradient descent is not the best optimizer for this problem. Adam optimizer would be a better option.

This type of neural network architecture can fundamentally not learn a function like sin(x).

  • 1
    $\begingroup$ Don't have a relu the last just before the output.. $\endgroup$ – Aditya Feb 23 '18 at 6:45

Since we know that sin(x) can have -ve values as well, Relu will kill all those mercilessly..

Also we can't have relu just before the output layer as it doesn't makes any sense..(Atleast I have never seen this architecture before..)

Also you can make that 100 drop to around 40

Check this link also


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.