1
$\begingroup$

I have a very simple feed forward neural network with keras that should learn a sinus. Why is the predictive power so bad and what is generally the best way to pinpoint issues with a network?

In the code below, I have one input neuron, 10 in the hidden layer, and one output. I would expect the network to perform much more accurately.

import numpy as np
from keras.layers import Dense, Activation
from keras.models import Sequential

x = np.arange(100)
y = np.sin(x)

model = Sequential([
    Dense(10, input_shape=(1,)),
    Activation('sigmoid'),
    Dense(1),
    Activation('sigmoid')
])

model.compile(loss='mean_squared_error', optimizer='SGD', metrics=['accuracy'])
model.fit(x, y, epochs=10, batch_size=1)

scores = model.evaluate(x, y, verbose=0)
print("Baseline Error: %.2f%%" % (100-scores[1]*100))

print(model.predict(np.array([.5])))

Output:

  1/100 [..............................] - ETA: 0s - loss: 1.2016 - acc: 0.0000e+00
 79/100 [======================>.......] - ETA: 0s - loss: 0.4665 - acc: 0.0127    
100/100 [==============================] - 0s - loss: 0.5044 - acc: 0.0100     
Baseline Error: 99.00%
[[ 0.35267803]]
$\endgroup$
2
  • $\begingroup$ Try more epochs $\endgroup$ May 31 '17 at 17:35
  • $\begingroup$ With 1000 epoches still no improvement: 100/100 [==============================] - 0s - loss: 0.4968 - acc: 0.0100 Baseline Error: 99.00% $\endgroup$
    – Nickpick
    May 31 '17 at 17:43
3
$\begingroup$

Accuracy is a metric meant for classification problems, look at the mean squared error instead. Your network is too small for a highly fluctuating function that you want to learn, if you divide your x by a smaller amount it would be easier to learn. Second of all, adding another layer and having an identity activation at the end will help quite a bit. Also taking batches bigger than 1 will make the gradient more stable. With a 1000 epochs I get to 0.00167 as mean squared error.

x = np.arange(200).reshape(-1,1) / 50
y = np.sin(x)

model = Sequential([
Dense(40, input_shape=(1,)),
Activation('sigmoid'),
Dense(12),
Activation('sigmoid'),
Dense(1)
    ])

model.compile(loss='mean_squared_error', optimizer='SGD', metrics=['mean_squared_error'])

for i in range(40):
    model.fit(x, y, nb_epoch=25, batch_size=8, verbose=0)
    predictions = model.predict(x)
    print(np.mean(np.square(predictions - y)))

The biggest issue is that the signal in your original small dataset is very difficult to learn, you can see this when you plot it, it will just collapse to the mean.

$\endgroup$
5
  • $\begingroup$ Not an unlikely scenario :) $\endgroup$ May 31 '17 at 18:43
  • $\begingroup$ ^ The OP might have thought np.sin worked in degrees, not radians (just noticed you did remove the sigmoid activation - this is critical because sigmoid output will never go negative, and half of OP's targets are negative) $\endgroup$ May 31 '17 at 18:53
  • $\begingroup$ Heh I didn't even think about that, I mixed it up with tanh for a second $\endgroup$ May 31 '17 at 19:02
  • $\begingroup$ Excellent point, there needs to be the possibility to output a negative number, I think this is the main issue $\endgroup$
    – Nickpick
    May 31 '17 at 22:01
  • 1
    $\begingroup$ Well your y is basically alternating in 100 steps, while it has only 10 nodes, it actually cannot learn to distinguish between all these things without a more complex model with more nodes and/or layers $\endgroup$ May 31 '17 at 22:08
10
$\begingroup$

Keep in mind that the Accuracy measure is measuring whether the values are Exactly The Same. I.e. it is a classification measure, whereas approximating a sin curve is much better suited for measurement as a regression problem.

That said:

In evaluating the network's performance, what is the network actually doing? Let's take the network and perform a little visual analysis on its performance:

import matplotlib.pyplot as plt
preds = model.predict(x)
plt.plot(x, y, 'b', x, preds, 'r--')
plt.ylabel('Y / Predicted Value')
plt.xlabel('X Value')
plt.show()

enter image description here

Hmm. The model seems to be minimizing error by simply getting closer and closer to guessing 0 for every value, rather than approximating the function. There are several hypotheses here to explain this. One is that the network is not complex enough to model the function. In order to test this, let's simplify the function- that is, let's bring the range down to one sine cycle:

x = np.arange(0, math.pi*2, 0.1)
y = np.sin(x)

And try to train the network again:

enter image description here

Not wonderful, but a better fit, certainly.

How about with 100 epochs instead of 10?

enter image description here

How about with 1000 epochs?

enter image description here

This is, of course, very interesting. After 1000 epochs, our networks is able to roughly approximate the downward curve from 1:0 ($\pi/2$: $\pi$) of the sine response, but not the initial upward curve 0:1 (0:$\pi/2$) or the region in which the function is negative ($\pi$:$2\pi$).

This result begs the question- what will it look like after 10000 epochs?

enter image description here

Not significantly better. It looks like we'll have to change the architecture of the network (more layers, more neurons, and/or different activation functions) to improve beyond this point.

To inform this architecture change, let's take a look at the sigmoid activation function:

enter image description here

Uh-oh. The value of this sigmoid function can only ever be in the range 0:1, and the range of the sin function is -1:1.

To correct this, let's just normalize the sin response between 0 and 1:

y = (np.sin(x)+1)/2 

Now, the network performs much better than before after 1000 epochs:

enter image description here

And 10000:

enter image description here

After 100000 epochs, it's roughly perfect:

enter image description here

Even still, this advancement doesn't help much on the larger sin range (after 1000 epochs):

x = np.arange(0, 100, 1)
y = (np.sin(x)+1)/2 

enter image description here

If we, however, take the model trained on a single sin curve and further train it on the larger range, we begin to see progress after 1000 epochs:

x = np.arange(0, 100, .1)
y = (np.sin(x)+1)/2 

model_copy = model
model_copy.fit(x, y, epochs=1000, batch_size=8, verbose=0)
model_copy_preds = model_copy.predict(x)

enter image description here

And more so after 10000 epochs:enter image description here

And more so, well into the third repetition after 100000 epochs:

enter image description here

So, with careful training, our network with a single layer of sigmoid activations appears to be learning to generalize the sin curve. Further investigation could find a limit to that generalization, certainly.

For reproduction:

import numpy as np
from keras.layers import Dense
from keras.models import Sequential
import matplotlib.pyplot as plt
import math

x = np.arange(0, math.pi*2, .1)
y = (np.sin(x)+1)/2 

model = Sequential([
    Dense(10, input_shape=(1,)),
    Activation('sigmoid'),
    Dense(1)
])

model.compile(loss='mean_squared_error', optimizer='SGD', metrics=['mean_squared_error'])
model.fit(x, y, epochs=100000, batch_size=8, verbose=0)

preds = model.predict(x)

plt.plot(x, y, 'b', x, preds, 'r--')
plt.ylabel('Y / Predicted Value')
plt.xlabel('X Value')
plt.show()

x = np.arange(0, 100, .1)
y = (np.sin(x)+1)/2 

model_copy = model
model_copy.fit(x, y, epochs=10000, batch_size=8, verbose=0)
model_copy_preds = model_copy.predict(x)

plt.plot(x, y, 'b', x, model_copy_preds, 'r--')
plt.ylabel('Y / Predicted Value')
plt.xlabel('X Value')
plt.show()
$\endgroup$
1
$\begingroup$

My code...

from keras.models import Sequential
from keras.layers import Dense
import numpy as np
import matplotlib.pylab as plt

# Create dataset
x = np.arange(0, np.pi * 2, 0.1)
y = np.sin(x)

# Some parameters
ACTIVE_FUN = 'tanh'
BATCH_SIZE = 1
VERBOSE=0

# Create the model
model = Sequential()
model.add(Dense(5, input_shape=(1,), activation=ACTIVE_FUN))
model.add(Dense(5, activation=ACTIVE_FUN))
model.add(Dense(1, activation='linear'))

# Compile the model
model.compile(loss='mean_squared_error', optimizer='sgd', metrics=['mean_squared_error'])

# Fit the model
model.fit(x, y, epochs=1000, batch_size=BATCH_SIZE, verbose=VERBOSE)

# Evaluate the model
scores = model.evaluate(x, y, verbose=VERBOSE)
print('%s: %.2f%%' % (model.metrics_names[1], scores[1] * 100))

# Make predictions
y_pred = model.predict(x)

# Plot
plt.plot(x, y, color='blue', linewidth=1, markersize='1')
plt.plot(x, y_pred, color='green', linewidth=1, markersize='1')
plt.xlabel('Angle [rad]')
plt.ylabel('sin(x)')
plt.axis('tight')
plt.show()
$\endgroup$
1
  • $\begingroup$ it would be helpful if you had some photos or some kind of information about how well this worked $\endgroup$
    – Supamee
    Dec 3 '18 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.