I have implemented a neural network with 1 hidden layer using sigmoid activation unit but after watching a video on how relu activation function can be much faster a tried implementing it but the cost function is either Nan or inf in python. I have found that many people have the same problem but I couldn't find any solution on the internet. I have added '#changed' comment to the lines that I have changed from the previous version in which I used sigmoid activation function. Related python code snippets:

def relu(arg):  #I have tried both relu and leaky relu
    return 1*(arg<0)*0.0001*arg + (arg>=0)*arg

def reluGrad(arg):
    for i in range(arg.shape[0]):
        for j in range(arg.shape[1]):
            if arg[i][j]>0:
    return arg
def softmax(x):
    x = x.transpose()
    e_x = np.exp(x - np.max(x))
    return (e_x / e_x.sum(axis=0)).transpose()

#forward prop:
a1 = np.insert(data,0,np.ones(len(data)),1).astype(np.float64)
    z2 = a1.dot(theta1)
    a2 = relu(z2) #changed
    a2 = np.insert(a2,0,np.ones(len(a2)),1)
    z3 = a2.dot(theta2)
    a3 = softmax(z3) #changed
#compute the cost:
    cost = -(output*(np.log(a3))+(1-output)*(np.log(1-a3))).sum()
    cost = (1/len(data))*cost + (lamb/(2*len(data)))*((np.delete(theta1,0,0)**2).sum() + (np.delete(theta2,0,0)**2).sum())

    sigma3 = a3-output
    sigma2 = (sigma3.dot(np.transpose(theta2)))* reluGrad(np.insert(z2,0,np.ones(len(z2)),1)) #changed
    sigma2 = np.delete(sigma2,0,1)
    delta2 = (np.transpose(a2)).dot(sigma3)
    delta1 = (np.transpose(a1)).dot(sigma2)

    grad1 = delta1/len(data) + (lamb/len(data))*np.insert(np.delete(theta1,0,0),0,np.zeros(len(theta1[0])),0)
    grad2 = delta2/len(data) + (lamb/len(data))*np.insert(np.delete(theta2,0,0),0,np.zeros(len(theta2[0])),0)

#update theta
    theta1 = theta1 - alpha*grad1
    theta2 = theta2 - alpha*grad2

What is causing this problem ? How can it be fixed ?


1 Answer 1


If a3 is 0 or 1, np.log(a3) or np.log(1-a3) is going to give an error as $\log(0)$ is undefined, and $\lim_{x -> 0} \log(x) = -\infty$

  • $\begingroup$ How can I fix this ? $\endgroup$
    – Saksham
    Jul 4, 2017 at 1:15
  • $\begingroup$ @AyushChaurasia I'm not even sure that is the problem, can you print out the log values to verify? Also see here: datascience.stackexchange.com/questions/9302/… for the suggestion to add a small value $10^{-15}$ in the log function so that it never reaches zero. $\endgroup$ Jul 4, 2017 at 6:32
  • $\begingroup$ it removed the error but now I am only getting 45-50% accuracy $\endgroup$
    – Saksham
    Jul 4, 2017 at 8:59
  • $\begingroup$ Training or validation error? If training, maybe your network is not big enough or needs more layers. $\endgroup$ Jul 4, 2017 at 19:44
  • $\begingroup$ @geometrkal i was talking about validation error. Every thing except the activation function is changed but accuracy has decreased dramatically . $\endgroup$
    – Saksham
    Jul 6, 2017 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.