2
$\begingroup$

I read a few tutorials on neural network backpropagation and decided to implement one from scratch. I tried to find this single error for the past few days I have in my code with no success.

I followed this tutorial in hopes of being able to implement a sine function approximator. This is a simple network: 1 input neuron, 10 hidden neurons and 1 output neuron. The activation function is sigmoid in the second layer. The exact same model easily works in Tensorflow.

def sigmoid(x):
    return 1 / (1 + np.math.e ** -x)


def sigmoid_deriv(x):
    return sigmoid(x) * (1 - sigmoid(x))


x_data = np.random.rand(500) * 15.0
y_data = [sin(x) for x in x_data]

ETA = .01

layer1 = 0
layer1_weights = np.random.rand(10) * 2. - 1.
layer2 = np.zeros(10)
layer2_weights = np.random.rand(10) * 2. - 1.
layer3 = 0

for loop_iter in range(500000):
    # data init
    index = np.random.randint(0, 500)
    x = x_data[index]
    y = y_data[index]

    # forward propagation
    # layer 1
    layer1 = x

    # layer 2
    layer2 = layer1_weights * layer1

    # layer 3
    layer3 = sum(sigmoid(layer2) * layer2_weights)

    # error
    error = .5 * (layer3 - y) ** 2  # L2 loss

    # backpropagation
    # error_wrt_layer3 * layer3_wrt_weights_layer2
    error_wrt_layer2_weights = (y - layer3) * sigmoid(layer2)
    # error_wrt_layer3 * layer3_wrt_out_layer2 * out_layer2_wrt_in_layer2 * in_layer2_wrt_weights_layer1
    error_wrt_layer1_weights = (y - layer3) * layer2_weights * sigmoid_deriv(sigmoid(layer2)) * layer1

    # update the weights
    layer2_weights -= ETA * error_wrt_layer2_weights
    layer1_weights -= ETA * error_wrt_layer1_weights

    if loop_iter % 10000 == 0:
        print(error)

The unexpected behavior is simply that the network doesn't converge. Please, review my error_wrt_... derivatives. The problem should be there.

Here's the Tensorflow code it works flawlessly with:

x_data = np.array(np.random.rand(500)).reshape(500, 1)
y_data = np.array([sin(x) for x in x_data]).reshape(500, 1)

x = tf.placeholder(tf.float32, shape=[None, 1])
y_true = tf.placeholder(tf.float32, shape=[None, 1])
W = tf.Variable(tf.random_uniform([1, 10], -1.0, 1.0))
hidden1 = tf.nn.sigmoid(tf.matmul(x, W))
W_hidden = tf.Variable(tf.random_uniform([10, 1], -1.0, 1.0))
output = tf.matmul(hidden1, W_hidden)

loss = tf.square(output - y_true) / 2.
optimizer = tf.train.GradientDescentOptimizer(.01)
train = optimizer.minimize(loss)

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

for i in range(500000):
    rand_index = np.random.randint(0, 500)
    _, error = sess.run([train, loss], feed_dict={x: [x_data[rand_index]],
                                              y_true: [y_data[rand_index]]})
    if i % 10000 == 0:
        print(error)

sess.close()
$\endgroup$
  • $\begingroup$ Note it is more usual to associate the sigmoid activation function with the layer before it (as a modifier to that layer's output), and not with the next layer (as a modifier to the layer's input). It is hard to read the code since you are relying a lot on the simple network structure to get scalar * vector terms (and no matrices), whilst most standard NN formulations will treat all inputs and outputs as vectors and weights as matrices. $\endgroup$ – Neil Slater Nov 7 '16 at 13:46
3
$\begingroup$

I think your biggest problem is the lack of biases. Between the input layer and the hidden layer, you should not only transform by the weights but should also add a bias. This bias will shift your sigmoid function to the left or right. Take a look at this code (I made some adaptations).

What is important:

  1. Added biases.
  2. Altered your error_w such that they are correct.
  3. Made some good random starting points for biases (np.random.rand(width) * 15. - 7.5) such that all biases are random points on the desired x-scale.
  4. Made a plot that shows the initial guess and final.

Let me know if some parts are not clear:

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(x):
    return 1 / (1 + np.math.e ** -x)


def sigmoid_deriv(x):
    return sigmoid(x) * (1 - sigmoid(x))

def guess(x):
    layer1 = x
    z_2 = layer1_weights * layer1 + layer1_biases
    a_2 =sigmoid(z_2)
    z_3 = np.dot(a_2, layer2_weights) + layer2_biases
    # a_3 = sigmoid(z_3)
    a_3 = z_3
    return a_3


x_data = np.random.rand(500) * 15.0 - 7.5
y_data = [np.sin(x) for x in x_data]

ETA = 0.05
width = 10

layer1_weights = np.random.rand(width) * 2. - 1.
layer1_biases = np.random.rand(width) * 15. - 7.5
layer2_weights = np.random.rand(width) * 2. - 1.
layer2_biases = np.random.rand(1)* 2. - 1.

error_all = []

x_all = x_data
y_all = [guess(x_i) for x_i in x_all]

plt.plot(x_all,y_all, '.')
plt.plot(x_data, y_data, '.')
plt.show()

epochs = 500000



for loop_iter in range(epochs):
    # data init
    index = np.random.randint(0, 500)
    x = x_data[index]
    y = y_data[index]

    # forward propagation
    # layer 1
    layer1 = x

    # layer 2
    #TODO add the sigmoid function here

    z_2 = layer1_weights * layer1 + layer1_biases
    a_2 =sigmoid(z_2)

    # layer 3
    #TODO remove simgmoid here (not that is really matters, but values at each layer are after sigmoid
    z_3 = np.dot(a_2, layer2_weights) + layer2_biases
    # a_3 = sigmoid(z_3)
    a_3 = z_3


    # error
    error = .5 * (a_3 - y) ** 2  # L2 loss

    # backpropagation
    # error_wrt_layer3 * layer3_wrt_weights_layer2

    # error_wrt_layer2_weights = (y - layer3) * sigmoid(layer2)

    delta = (a_3 - y)

    error_wrt_layer2_weights = delta * a_2
    error_wrt_layer2_biases = delta

    # error_wrt_layer3 * layer3_wrt_out_layer2 * out_layer2_wrt_in_layer2 * in_layer2_wrt_weights_layer1

    # error_wrt_layer1_weights = (y - layer3) * layer2_weights * sigmoid_deriv(sigmoid(layer2)) * layer1
    error_wrt_layer1_weights = delta * np.dot(sigmoid_deriv(z_2), layer2_weights) * layer1
    # error_wrt_layer1_weights = 0

    error_wrt_layer1_biases =  delta * np.dot(sigmoid_deriv(z_2), layer2_weights)


    # a = 0
    # while a ==0:
    #   a*0

    # update the weights
    layer2_weights -= ETA * error_wrt_layer2_weights
    layer1_weights -= ETA * error_wrt_layer1_weights
    layer2_biases -= ETA * error_wrt_layer2_biases
    layer1_biases -= ETA * error_wrt_layer1_biases

    error_all.append(error)

    if loop_iter % 10000 == 0:
        print(error)



# plt.plot(error_all)
# plt.show()

x_all = x_data
y_all = [guess(x_i) for x_i in x_all]

plt.plot(x_all,y_all, '.')
plt.plot(x_data, y_data, '.')
plt.show()
| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for your comment. It's better, but I still get a ~0.75 error occasionally. It vexes me how Tensorflow gets an error ~0.0006 after just 2000 steps without biases. Isn't it Gradient Descent too? $\endgroup$ – icancto Nov 7 '16 at 14:57
  • 2
    $\begingroup$ @icancto: Probably worth showing your TensorFlow implementation. Even the basic gradient descent optimiser in TF has momentum param - if you use that then it will converge faster and more robustly than your implementation even if everything else is the same. $\endgroup$ – Neil Slater Nov 7 '16 at 15:15
  • $\begingroup$ @NeilSlater Sure thing, added it! $\endgroup$ – icancto Nov 7 '16 at 15:28
  • $\begingroup$ But Tensorflow only checks values between 0 and 1 right? This is much easier since sin(x) with x between 0 and 1 is Not a tricky curve. Rerun the program with x_data = np.random.rand(500) and you will get very nice results ;). I now get an error of 10E-7 ofcourse with biases, but this is an insane increase in performance $\endgroup$ – Laurens Meeus Nov 7 '16 at 15:40
  • $\begingroup$ @LaurensMeeus Great! I might have calculated the derivatives correctly then. Oh well. What would be some additional steps to reduce the error further? $\endgroup$ – icancto Nov 7 '16 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.