Backpropagation derivation problem

Question

I read a few tutorials on neural network backpropagation and decided to implement one from scratch. I tried to find this single error for the past few days I have in my code with no success.

I followed this tutorial in hopes of being able to implement a sine function approximator. This is a simple network: 1 input neuron, 10 hidden neurons and 1 output neuron. The activation function is sigmoid in the second layer. The exact same model easily works in Tensorflow.

def sigmoid(x):
    return 1 / (1 + np.math.e ** -x)


def sigmoid_deriv(x):
    return sigmoid(x) * (1 - sigmoid(x))


x_data = np.random.rand(500) * 15.0
y_data = [sin(x) for x in x_data]

ETA = .01

layer1 = 0
layer1_weights = np.random.rand(10) * 2. - 1.
layer2 = np.zeros(10)
layer2_weights = np.random.rand(10) * 2. - 1.
layer3 = 0

for loop_iter in range(500000):
    # data init
    index = np.random.randint(0, 500)
    x = x_data[index]
    y = y_data[index]

    # forward propagation
    # layer 1
    layer1 = x

    # layer 2
    layer2 = layer1_weights * layer1

    # layer 3
    layer3 = sum(sigmoid(layer2) * layer2_weights)

    # error
    error = .5 * (layer3 - y) ** 2  # L2 loss

    # backpropagation
    # error_wrt_layer3 * layer3_wrt_weights_layer2
    error_wrt_layer2_weights = (y - layer3) * sigmoid(layer2)
    # error_wrt_layer3 * layer3_wrt_out_layer2 * out_layer2_wrt_in_layer2 * in_layer2_wrt_weights_layer1
    error_wrt_layer1_weights = (y - layer3) * layer2_weights * sigmoid_deriv(sigmoid(layer2)) * layer1

    # update the weights
    layer2_weights -= ETA * error_wrt_layer2_weights
    layer1_weights -= ETA * error_wrt_layer1_weights

    if loop_iter % 10000 == 0:
        print(error)

The unexpected behavior is simply that the network doesn't converge. Please, review my error_wrt_... derivatives. The problem should be there.

Here's the Tensorflow code it works flawlessly with:

x_data = np.array(np.random.rand(500)).reshape(500, 1)
y_data = np.array([sin(x) for x in x_data]).reshape(500, 1)

x = tf.placeholder(tf.float32, shape=[None, 1])
y_true = tf.placeholder(tf.float32, shape=[None, 1])
W = tf.Variable(tf.random_uniform([1, 10], -1.0, 1.0))
hidden1 = tf.nn.sigmoid(tf.matmul(x, W))
W_hidden = tf.Variable(tf.random_uniform([10, 1], -1.0, 1.0))
output = tf.matmul(hidden1, W_hidden)

loss = tf.square(output - y_true) / 2.
optimizer = tf.train.GradientDescentOptimizer(.01)
train = optimizer.minimize(loss)

init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)

for i in range(500000):
    rand_index = np.random.randint(0, 500)
    _, error = sess.run([train, loss], feed_dict={x: [x_data[rand_index]],
                                              y_true: [y_data[rand_index]]})
    if i % 10000 == 0:
        print(error)

sess.close()

Answer 1

I think your biggest problem is the lack of biases. Between the input layer and the hidden layer, you should not only transform by the weights but should also add a bias. This bias will shift your sigmoid function to the left or right. Take a look at this code (I made some adaptations).

What is important:

1. Added biases.
2. Altered your error_w such that they are correct.
3. Made some good random starting points for biases (np.random.rand(width) * 15. - 7.5) such that all biases are random points on the desired x-scale.
4. Made a plot that shows the initial guess and final.

Let me know if some parts are not clear:

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(x):
    return 1 / (1 + np.math.e ** -x)


def sigmoid_deriv(x):
    return sigmoid(x) * (1 - sigmoid(x))

def guess(x):
    layer1 = x
    z_2 = layer1_weights * layer1 + layer1_biases
    a_2 =sigmoid(z_2)
    z_3 = np.dot(a_2, layer2_weights) + layer2_biases
    # a_3 = sigmoid(z_3)
    a_3 = z_3
    return a_3


x_data = np.random.rand(500) * 15.0 - 7.5
y_data = [np.sin(x) for x in x_data]

ETA = 0.05
width = 10

layer1_weights = np.random.rand(width) * 2. - 1.
layer1_biases = np.random.rand(width) * 15. - 7.5
layer2_weights = np.random.rand(width) * 2. - 1.
layer2_biases = np.random.rand(1)* 2. - 1.

error_all = []

x_all = x_data
y_all = [guess(x_i) for x_i in x_all]

plt.plot(x_all,y_all, '.')
plt.plot(x_data, y_data, '.')
plt.show()

epochs = 500000



for loop_iter in range(epochs):
    # data init
    index = np.random.randint(0, 500)
    x = x_data[index]
    y = y_data[index]

    # forward propagation
    # layer 1
    layer1 = x

    # layer 2
    #TODO add the sigmoid function here

    z_2 = layer1_weights * layer1 + layer1_biases
    a_2 =sigmoid(z_2)

    # layer 3
    #TODO remove simgmoid here (not that is really matters, but values at each layer are after sigmoid
    z_3 = np.dot(a_2, layer2_weights) + layer2_biases
    # a_3 = sigmoid(z_3)
    a_3 = z_3


    # error
    error = .5 * (a_3 - y) ** 2  # L2 loss

    # backpropagation
    # error_wrt_layer3 * layer3_wrt_weights_layer2

    # error_wrt_layer2_weights = (y - layer3) * sigmoid(layer2)

    delta = (a_3 - y)

    error_wrt_layer2_weights = delta * a_2
    error_wrt_layer2_biases = delta

    # error_wrt_layer3 * layer3_wrt_out_layer2 * out_layer2_wrt_in_layer2 * in_layer2_wrt_weights_layer1

    # error_wrt_layer1_weights = (y - layer3) * layer2_weights * sigmoid_deriv(sigmoid(layer2)) * layer1
    error_wrt_layer1_weights = delta * np.dot(sigmoid_deriv(z_2), layer2_weights) * layer1
    # error_wrt_layer1_weights = 0

    error_wrt_layer1_biases =  delta * np.dot(sigmoid_deriv(z_2), layer2_weights)


    # a = 0
    # while a ==0:
    #   a*0

    # update the weights
    layer2_weights -= ETA * error_wrt_layer2_weights
    layer1_weights -= ETA * error_wrt_layer1_weights
    layer2_biases -= ETA * error_wrt_layer2_biases
    layer1_biases -= ETA * error_wrt_layer1_biases

    error_all.append(error)

    if loop_iter % 10000 == 0:
        print(error)



# plt.plot(error_all)
# plt.show()

x_all = x_data
y_all = [guess(x_i) for x_i in x_all]

plt.plot(x_all,y_all, '.')
plt.plot(x_data, y_data, '.')
plt.show()