0
$\begingroup$

I have implemented a neural network with 1 hidden layer using sigmoid activation unit but after watching a video on how relu activation function can be much faster a tried implementing it but the cost function is either Nan or inf in python. I have found that many people have the same problem but I couldn't find any solution on the internet. I have added '#changed' comment to the lines that I have changed from the previous version in which I used sigmoid activation function. Related python code snippets:

def relu(arg):  #I have tried both relu and leaky relu
    return 1*(arg<0)*0.0001*arg + (arg>=0)*arg

def reluGrad(arg):
    for i in range(arg.shape[0]):
        for j in range(arg.shape[1]):
            if arg[i][j]>0:
                arg[i][j]=1
            else:
                arg[i][j]=0
    return arg
def softmax(x):
    x = x.transpose()
    e_x = np.exp(x - np.max(x))
    return (e_x / e_x.sum(axis=0)).transpose()

#forward prop:
a1 = np.insert(data,0,np.ones(len(data)),1).astype(np.float64)
    z2 = a1.dot(theta1)
    a2 = relu(z2) #changed
    a2 = np.insert(a2,0,np.ones(len(a2)),1)
    z3 = a2.dot(theta2)
    a3 = softmax(z3) #changed
#compute the cost:
    cost = -(output*(np.log(a3))+(1-output)*(np.log(1-a3))).sum()
    cost = (1/len(data))*cost + (lamb/(2*len(data)))*((np.delete(theta1,0,0)**2).sum() + (np.delete(theta2,0,0)**2).sum())

backProp:
    sigma3 = a3-output
    sigma2 = (sigma3.dot(np.transpose(theta2)))* reluGrad(np.insert(z2,0,np.ones(len(z2)),1)) #changed
    sigma2 = np.delete(sigma2,0,1)
    delta2 = (np.transpose(a2)).dot(sigma3)
    delta1 = (np.transpose(a1)).dot(sigma2)

    grad1 = delta1/len(data) + (lamb/len(data))*np.insert(np.delete(theta1,0,0),0,np.zeros(len(theta1[0])),0)
    grad2 = delta2/len(data) + (lamb/len(data))*np.insert(np.delete(theta2,0,0),0,np.zeros(len(theta2[0])),0)

#update theta
    theta1 = theta1 - alpha*grad1
    theta2 = theta2 - alpha*grad2

What is causing this problem ? How can it be fixed ?

$\endgroup$
1
$\begingroup$

If a3 is 0 or 1, np.log(a3) or np.log(1-a3) is going to give an error as $\log(0)$ is undefined, and $\lim_{x -> 0} \log(x) = -\infty$

|improve this answer|||||
$\endgroup$
  • $\begingroup$ How can I fix this ? $\endgroup$ – Saksham Jul 4 '17 at 1:15
  • $\begingroup$ @AyushChaurasia I'm not even sure that is the problem, can you print out the log values to verify? Also see here: datascience.stackexchange.com/questions/9302/… for the suggestion to add a small value $10^{-15}$ in the log function so that it never reaches zero. $\endgroup$ – geometrikal Jul 4 '17 at 6:32
  • $\begingroup$ it removed the error but now I am only getting 45-50% accuracy $\endgroup$ – Saksham Jul 4 '17 at 8:59
  • $\begingroup$ Training or validation error? If training, maybe your network is not big enough or needs more layers. $\endgroup$ – geometrikal Jul 4 '17 at 19:44
  • $\begingroup$ @geometrkal i was talking about validation error. Every thing except the activation function is changed but accuracy has decreased dramatically . $\endgroup$ – Saksham Jul 6 '17 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.