Backpropagation derivation problem

I read a few tutorials on neural network backpropagation and decided to implement one from scratch. I tried to find this single error for the past few days I have in my code with no success.

I followed this tutorial in hopes of being able to implement a sine function approximator. This is a simple network: 1 input neuron, 10 hidden neurons and 1 output neuron. The activation function is sigmoid in the second layer. The exact same model easily works in Tensorflow.

def sigmoid(x):
return 1 / (1 + np.math.e ** -x)

def sigmoid_deriv(x):
return sigmoid(x) * (1 - sigmoid(x))

x_data = np.random.rand(500) * 15.0
y_data = [sin(x) for x in x_data]

ETA = .01

layer1 = 0
layer1_weights = np.random.rand(10) * 2. - 1.
layer2 = np.zeros(10)
layer2_weights = np.random.rand(10) * 2. - 1.
layer3 = 0

for loop_iter in range(500000):
# data init
index = np.random.randint(0, 500)
x = x_data[index]
y = y_data[index]

# forward propagation
# layer 1
layer1 = x

# layer 2
layer2 = layer1_weights * layer1

# layer 3
layer3 = sum(sigmoid(layer2) * layer2_weights)

# error
error = .5 * (layer3 - y) ** 2  # L2 loss

# backpropagation
# error_wrt_layer3 * layer3_wrt_weights_layer2
error_wrt_layer2_weights = (y - layer3) * sigmoid(layer2)
# error_wrt_layer3 * layer3_wrt_out_layer2 * out_layer2_wrt_in_layer2 * in_layer2_wrt_weights_layer1
error_wrt_layer1_weights = (y - layer3) * layer2_weights * sigmoid_deriv(sigmoid(layer2)) * layer1

# update the weights
layer2_weights -= ETA * error_wrt_layer2_weights
layer1_weights -= ETA * error_wrt_layer1_weights

if loop_iter % 10000 == 0:
print(error)


The unexpected behavior is simply that the network doesn't converge. Please, review my error_wrt_... derivatives. The problem should be there.

Here's the Tensorflow code it works flawlessly with:

x_data = np.array(np.random.rand(500)).reshape(500, 1)
y_data = np.array([sin(x) for x in x_data]).reshape(500, 1)

x = tf.placeholder(tf.float32, shape=[None, 1])
y_true = tf.placeholder(tf.float32, shape=[None, 1])
W = tf.Variable(tf.random_uniform([1, 10], -1.0, 1.0))
hidden1 = tf.nn.sigmoid(tf.matmul(x, W))
W_hidden = tf.Variable(tf.random_uniform([10, 1], -1.0, 1.0))
output = tf.matmul(hidden1, W_hidden)

loss = tf.square(output - y_true) / 2.
train = optimizer.minimize(loss)

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

for i in range(500000):
rand_index = np.random.randint(0, 500)
_, error = sess.run([train, loss], feed_dict={x: [x_data[rand_index]],
y_true: [y_data[rand_index]]})
if i % 10000 == 0:
print(error)

sess.close()

• Note it is more usual to associate the sigmoid activation function with the layer before it (as a modifier to that layer's output), and not with the next layer (as a modifier to the layer's input). It is hard to read the code since you are relying a lot on the simple network structure to get scalar * vector terms (and no matrices), whilst most standard NN formulations will treat all inputs and outputs as vectors and weights as matrices. – Neil Slater Nov 7 '16 at 13:46

I think your biggest problem is the lack of biases. Between the input layer and the hidden layer, you should not only transform by the weights but should also add a bias. This bias will shift your sigmoid function to the left or right. Take a look at this code (I made some adaptations).

What is important:

2. Altered your error_w such that they are correct.
3. Made some good random starting points for biases (np.random.rand(width) * 15. - 7.5) such that all biases are random points on the desired x-scale.
4. Made a plot that shows the initial guess and final.

Let me know if some parts are not clear:

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(x):
return 1 / (1 + np.math.e ** -x)

def sigmoid_deriv(x):
return sigmoid(x) * (1 - sigmoid(x))

def guess(x):
layer1 = x
z_2 = layer1_weights * layer1 + layer1_biases
a_2 =sigmoid(z_2)
z_3 = np.dot(a_2, layer2_weights) + layer2_biases
# a_3 = sigmoid(z_3)
a_3 = z_3
return a_3

x_data = np.random.rand(500) * 15.0 - 7.5
y_data = [np.sin(x) for x in x_data]

ETA = 0.05
width = 10

layer1_weights = np.random.rand(width) * 2. - 1.
layer1_biases = np.random.rand(width) * 15. - 7.5
layer2_weights = np.random.rand(width) * 2. - 1.
layer2_biases = np.random.rand(1)* 2. - 1.

error_all = []

x_all = x_data
y_all = [guess(x_i) for x_i in x_all]

plt.plot(x_all,y_all, '.')
plt.plot(x_data, y_data, '.')
plt.show()

epochs = 500000

for loop_iter in range(epochs):
# data init
index = np.random.randint(0, 500)
x = x_data[index]
y = y_data[index]

# forward propagation
# layer 1
layer1 = x

# layer 2
#TODO add the sigmoid function here

z_2 = layer1_weights * layer1 + layer1_biases
a_2 =sigmoid(z_2)

# layer 3
#TODO remove simgmoid here (not that is really matters, but values at each layer are after sigmoid
z_3 = np.dot(a_2, layer2_weights) + layer2_biases
# a_3 = sigmoid(z_3)
a_3 = z_3

# error
error = .5 * (a_3 - y) ** 2  # L2 loss

# backpropagation
# error_wrt_layer3 * layer3_wrt_weights_layer2

# error_wrt_layer2_weights = (y - layer3) * sigmoid(layer2)

delta = (a_3 - y)

error_wrt_layer2_weights = delta * a_2
error_wrt_layer2_biases = delta

# error_wrt_layer3 * layer3_wrt_out_layer2 * out_layer2_wrt_in_layer2 * in_layer2_wrt_weights_layer1

# error_wrt_layer1_weights = (y - layer3) * layer2_weights * sigmoid_deriv(sigmoid(layer2)) * layer1
error_wrt_layer1_weights = delta * np.dot(sigmoid_deriv(z_2), layer2_weights) * layer1
# error_wrt_layer1_weights = 0

error_wrt_layer1_biases =  delta * np.dot(sigmoid_deriv(z_2), layer2_weights)

# a = 0
# while a ==0:
#   a*0

# update the weights
layer2_weights -= ETA * error_wrt_layer2_weights
layer1_weights -= ETA * error_wrt_layer1_weights
layer2_biases -= ETA * error_wrt_layer2_biases
layer1_biases -= ETA * error_wrt_layer1_biases

error_all.append(error)

if loop_iter % 10000 == 0:
print(error)

# plt.plot(error_all)
# plt.show()

x_all = x_data
y_all = [guess(x_i) for x_i in x_all]

plt.plot(x_all,y_all, '.')
plt.plot(x_data, y_data, '.')
plt.show()

• Thank you for your comment. It's better, but I still get a ~0.75 error occasionally. It vexes me how Tensorflow gets an error ~0.0006 after just 2000 steps without biases. Isn't it Gradient Descent too? – icancto Nov 7 '16 at 14:57
• @icancto: Probably worth showing your TensorFlow implementation. Even the basic gradient descent optimiser in TF has momentum param - if you use that then it will converge faster and more robustly than your implementation even if everything else is the same. – Neil Slater Nov 7 '16 at 15:15
• @NeilSlater Sure thing, added it! – icancto Nov 7 '16 at 15:28
• But Tensorflow only checks values between 0 and 1 right? This is much easier since sin(x) with x between 0 and 1 is Not a tricky curve. Rerun the program with x_data = np.random.rand(500) and you will get very nice results ;). I now get an error of 10E-7 ofcourse with biases, but this is an insane increase in performance – Laurens Meeus Nov 7 '16 at 15:40
• @LaurensMeeus Great! I might have calculated the derivatives correctly then. Oh well. What would be some additional steps to reduce the error further? – icancto Nov 7 '16 at 15:50