I am having a problem understanding the cost function in a neural network. I have read many books and blog posts, but all of them describe that point in neural networks is to minimize the cost function (like sum squared error):

enter image description here

I tried to look at code for solving a problem with a multi layer neural network and back propagation. My question is: where in the code can I find the cost function? How can I plot the error surface?

import numpy as np
X_XOR = np.array([[0,0,1], [0,1,1], [1,0,1],[1,1,1]])
y_truth = np.array([[0],[1],[1],[0]])

def sigmoid(x):
    return 1 / (1 + np.exp(-x))
def sigmoid_der(output):
    return output * (1 - output)

syn_0 = 2*np.random.random((3,4)) - 1
syn_1 = 2*np.random.random((4,1)) - 1

for i in range(60000):
    layer_1 = sigmoid(X_XOR.dot(syn_0))
    layer_2 = sigmoid(layer_1.dot(syn_1))
    error = 0.5 * ((layer_2 - y_truth) ** 2)
    layer_2_delta = error * sigmoid_der(layer_2)
    layer_1_error = layer_2_delta.dot(syn_1.T)
    layer_1_delta = layer_1_error * sigmoid_der(layer_1)
    syn_1 -= layer_1.T.dot(layer_2_delta)
    syn_0 -= X_XOR.T.dot(layer_1_delta)
    if i % 10000 == 1:

  • $\begingroup$ @JahKnows can you explain? $\endgroup$
    – lukassz
    May 16 '18 at 17:07

The cost function can be found in the delta rule, meaning the way you calculate your deltas. This delta is nothing more than the derivative of your error function after the weights: $\frac{\partial E}{\partial w_{ij}}$. So, if you are just interested in where the cost is encoded, this is the answer you are looking for.

If you, on the other hand want to know why this formula works, I can suggest you to read the derivation on wikipedia. The maths behind it is quite uncomplicated, you only compute the derivative in each layer and propagate this derivative through the layers. This is, by the way, how backpropagation got its name.

  • $\begingroup$ Ok, It possible to plot error surface for my solution? $\endgroup$
    – lukassz
    May 17 '18 at 10:22
  • $\begingroup$ You can simply plot your error surface by plotting the variable "error" in each step of your algorithm, if that is what you desire. $\endgroup$
    – André
    May 20 '18 at 13:06
  • $\begingroup$ not exactly, I want to plot 3d error surface $\endgroup$
    – lukassz
    May 22 '18 at 14:33
  • $\begingroup$ Can you please specify what you want to plot. I was thinking about the iterations on the x axis and the error on the y axis. $\endgroup$
    – André
    May 22 '18 at 16:13
  • $\begingroup$ My neural network has 2 layer. I was thinking about 3d error surface like this imgur.com/60Evv6U $\endgroup$
    – lukassz
    May 22 '18 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.