# Predict sinus with keras feed forward neural network

I have a very simple feed forward neural network with keras that should learn a sinus. Why is the predictive power so bad and what is generally the best way to pinpoint issues with a network?

In the code below, I have one input neuron, 10 in the hidden layer, and one output. I would expect the network to perform much more accurately.

import numpy as np
from keras.layers import Dense, Activation
from keras.models import Sequential

x = np.arange(100)
y = np.sin(x)

model = Sequential([
Dense(10, input_shape=(1,)),
Activation('sigmoid'),
Dense(1),
Activation('sigmoid')
])

model.compile(loss='mean_squared_error', optimizer='SGD', metrics=['accuracy'])
model.fit(x, y, epochs=10, batch_size=1)

scores = model.evaluate(x, y, verbose=0)
print("Baseline Error: %.2f%%" % (100-scores[1]*100))

print(model.predict(np.array([.5])))


Output:

  1/100 [..............................] - ETA: 0s - loss: 1.2016 - acc: 0.0000e+00
79/100 [======================>.......] - ETA: 0s - loss: 0.4665 - acc: 0.0127
100/100 [==============================] - 0s - loss: 0.5044 - acc: 0.0100
Baseline Error: 99.00%
[[ 0.35267803]]

• Try more epochs May 31 '17 at 17:35
• With 1000 epoches still no improvement: 100/100 [==============================] - 0s - loss: 0.4968 - acc: 0.0100 Baseline Error: 99.00% May 31 '17 at 17:43

Accuracy is a metric meant for classification problems, look at the mean squared error instead. Your network is too small for a highly fluctuating function that you want to learn, if you divide your x by a smaller amount it would be easier to learn. Second of all, adding another layer and having an identity activation at the end will help quite a bit. Also taking batches bigger than 1 will make the gradient more stable. With a 1000 epochs I get to 0.00167 as mean squared error.

x = np.arange(200).reshape(-1,1) / 50
y = np.sin(x)

model = Sequential([
Dense(40, input_shape=(1,)),
Activation('sigmoid'),
Dense(12),
Activation('sigmoid'),
Dense(1)
])

model.compile(loss='mean_squared_error', optimizer='SGD', metrics=['mean_squared_error'])

for i in range(40):
model.fit(x, y, nb_epoch=25, batch_size=8, verbose=0)
predictions = model.predict(x)
print(np.mean(np.square(predictions - y)))


The biggest issue is that the signal in your original small dataset is very difficult to learn, you can see this when you plot it, it will just collapse to the mean.

• Not an unlikely scenario :) May 31 '17 at 18:43
• ^ The OP might have thought np.sin worked in degrees, not radians (just noticed you did remove the sigmoid activation - this is critical because sigmoid output will never go negative, and half of OP's targets are negative) May 31 '17 at 18:53
• Heh I didn't even think about that, I mixed it up with tanh for a second May 31 '17 at 19:02
• Excellent point, there needs to be the possibility to output a negative number, I think this is the main issue May 31 '17 at 22:01
• Well your y is basically alternating in 100 steps, while it has only 10 nodes, it actually cannot learn to distinguish between all these things without a more complex model with more nodes and/or layers May 31 '17 at 22:08

Keep in mind that the Accuracy measure is measuring whether the values are Exactly The Same. I.e. it is a classification measure, whereas approximating a sin curve is much better suited for measurement as a regression problem.

That said:

In evaluating the network's performance, what is the network actually doing? Let's take the network and perform a little visual analysis on its performance:

import matplotlib.pyplot as plt
preds = model.predict(x)
plt.plot(x, y, 'b', x, preds, 'r--')
plt.ylabel('Y / Predicted Value')
plt.xlabel('X Value')
plt.show()


Hmm. The model seems to be minimizing error by simply getting closer and closer to guessing 0 for every value, rather than approximating the function. There are several hypotheses here to explain this. One is that the network is not complex enough to model the function. In order to test this, let's simplify the function- that is, let's bring the range down to one sine cycle:

x = np.arange(0, math.pi*2, 0.1)
y = np.sin(x)


And try to train the network again:

Not wonderful, but a better fit, certainly.

This is, of course, very interesting. After 1000 epochs, our networks is able to roughly approximate the downward curve from 1:0 ($\pi/2$: $\pi$) of the sine response, but not the initial upward curve 0:1 (0:$\pi/2$) or the region in which the function is negative ($\pi$:$2\pi$).

This result begs the question- what will it look like after 10000 epochs?

Not significantly better. It looks like we'll have to change the architecture of the network (more layers, more neurons, and/or different activation functions) to improve beyond this point.

To inform this architecture change, let's take a look at the sigmoid activation function:

Uh-oh. The value of this sigmoid function can only ever be in the range 0:1, and the range of the sin function is -1:1.

To correct this, let's just normalize the sin response between 0 and 1:

y = (np.sin(x)+1)/2


Now, the network performs much better than before after 1000 epochs:

And 10000:

After 100000 epochs, it's roughly perfect:

Even still, this advancement doesn't help much on the larger sin range (after 1000 epochs):

x = np.arange(0, 100, 1)
y = (np.sin(x)+1)/2


If we, however, take the model trained on a single sin curve and further train it on the larger range, we begin to see progress after 1000 epochs:

x = np.arange(0, 100, .1)
y = (np.sin(x)+1)/2

model_copy = model
model_copy.fit(x, y, epochs=1000, batch_size=8, verbose=0)
model_copy_preds = model_copy.predict(x)


And more so after 10000 epochs:

And more so, well into the third repetition after 100000 epochs:

So, with careful training, our network with a single layer of sigmoid activations appears to be learning to generalize the sin curve. Further investigation could find a limit to that generalization, certainly.

For reproduction:

import numpy as np
from keras.layers import Dense
from keras.models import Sequential
import matplotlib.pyplot as plt
import math

x = np.arange(0, math.pi*2, .1)
y = (np.sin(x)+1)/2

model = Sequential([
Dense(10, input_shape=(1,)),
Activation('sigmoid'),
Dense(1)
])

model.compile(loss='mean_squared_error', optimizer='SGD', metrics=['mean_squared_error'])
model.fit(x, y, epochs=100000, batch_size=8, verbose=0)

preds = model.predict(x)

plt.plot(x, y, 'b', x, preds, 'r--')
plt.ylabel('Y / Predicted Value')
plt.xlabel('X Value')
plt.show()

x = np.arange(0, 100, .1)
y = (np.sin(x)+1)/2

model_copy = model
model_copy.fit(x, y, epochs=10000, batch_size=8, verbose=0)
model_copy_preds = model_copy.predict(x)

plt.plot(x, y, 'b', x, model_copy_preds, 'r--')
plt.ylabel('Y / Predicted Value')
plt.xlabel('X Value')
plt.show()


My code...

from keras.models import Sequential
from keras.layers import Dense
import numpy as np
import matplotlib.pylab as plt

# Create dataset
x = np.arange(0, np.pi * 2, 0.1)
y = np.sin(x)

# Some parameters
ACTIVE_FUN = 'tanh'
BATCH_SIZE = 1
VERBOSE=0

# Create the model
model = Sequential()

# Compile the model
model.compile(loss='mean_squared_error', optimizer='sgd', metrics=['mean_squared_error'])

# Fit the model
model.fit(x, y, epochs=1000, batch_size=BATCH_SIZE, verbose=VERBOSE)

# Evaluate the model
scores = model.evaluate(x, y, verbose=VERBOSE)
print('%s: %.2f%%' % (model.metrics_names[1], scores[1] * 100))

# Make predictions
y_pred = model.predict(x)

# Plot
plt.plot(x, y, color='blue', linewidth=1, markersize='1')
plt.plot(x, y_pred, color='green', linewidth=1, markersize='1')