1
$\begingroup$

Below is the code by Siraj Raval for implementing Neural Networks from Scratch. I have some doubts regarding the Code:

  1. Why during updation he did W2 = W2 + L1.T.dot(L2_Delta). I Mean shouldn't it be W2 = W2 - L1.T.dot(L2_Delta) as we should move in the opposite direction of the gradient as in Gradient Descent

  2. According to the fundamental equation of Backpropagation

    enter image description here

The Update should be done as Shown in equation BP2 but in code, but the order is opposite in the code ??

L1_Delta =(L2_Delta.dot(W2.T)) * Sigmoid(L1,Deriv = True) # Hidden Layer Error

Similarly for Equation BP4 the Gradient

W2 += L1.T.dot(L2_Delta)

I think It Should be :

W2 += (L2_Delta).dot(L1.T)

# A Three Layer Model Layer 1: Input Layer
# Layer 2 = Hidden layer , Layer 3 = Output layer
#nO.OF wEIGHTS = #lAYERS - 1
import numpy as np

def Sigmoid(Z,Deriv = False):
    P = np.reciprocal(1+np.exp(-Z))
    if Deriv == False:
        return P
    else:
        return Z*(1-Z)

#Input Data
X = np.array([[0,0,1],[0,1,1],[1,0,1],[1,1,1]])
Y = np.array([[0],
              [1],
              [1],
              [0]])

np.random.seed(1)
W1 = 2*np.random.random((3,4)) - 1
W2 = 2*np.random.random((4,1)) - 1

#Training

for j in range(60000):
    L0 = X  #Input Layer
    L1 = Sigmoid(np.dot(L0,W1))  #First Hidden Layer
    L2 = Sigmoid(np.dot(L1,W2))  # Output Layer

    L2_Error = Y - L2

    if j % 1000 == 0:
        print("Error: " + str(np.mean(np.abs(L2_Error))))

    L2_Delta = L2_Error * Sigmoid(L2,Deriv = True)   #Output Layer Error
    #Now How much layer 1 contributed in the error of Layer 2
    L1_Error = L2_Delta.dot(W2.T)
    L1_Delta = L1_Error * Sigmoid(L1,Deriv = True)  # Hidden Layer Erro

    #Updates
    W2 += L1.T.dot(L2_Delta)
    W1 += L0.T.dot(L1_Delta)
print("Output After Training")
print(L2)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.