XOR problem with neural network, cost function

Question

I am having a problem understanding the cost function in a neural network. I have read many books and blog posts, but all of them describe that point in neural networks is to minimize the cost function (like sum squared error):

I tried to look at code for solving a problem with a multi layer neural network and back propagation. My question is: where in the code can I find the cost function? How can I plot the error surface?

import numpy as np
X_XOR = np.array([[0,0,1], [0,1,1], [1,0,1],[1,1,1]])
y_truth = np.array([[0],[1],[1],[0]])

def sigmoid(x):
    return 1 / (1 + np.exp(-x))
def sigmoid_der(output):
    return output * (1 - output)

np.random.seed(1)
syn_0 = 2*np.random.random((3,4)) - 1
syn_1 = 2*np.random.random((4,1)) - 1

for i in range(60000):
    layer_1 = sigmoid(X_XOR.dot(syn_0))
    layer_2 = sigmoid(layer_1.dot(syn_1))
    error = 0.5 * ((layer_2 - y_truth) ** 2)
    layer_2_delta = error * sigmoid_der(layer_2)
    layer_1_error = layer_2_delta.dot(syn_1.T)
    layer_1_delta = layer_1_error * sigmoid_der(layer_1)
    syn_1 -= layer_1.T.dot(layer_2_delta)
    syn_0 -= X_XOR.T.dot(layer_1_delta)
    if i % 10000 == 1:
        print(layer_2)

print(layer_2)

Answer 1

The cost function can be found in the delta rule, meaning the way you calculate your deltas. This delta is nothing more than the derivative of your error function after the weights: $\frac{\partial E}{\partial w_{ij}}$. So, if you are just interested in where the cost is encoded, this is the answer you are looking for.

If you, on the other hand want to know why this formula works, I can suggest you to read the derivation on wikipedia. The maths behind it is quite uncomplicated, you only compute the derivative in each layer and propagate this derivative through the layers. This is, by the way, how backpropagation got its name.