Can ReLU replace a Sigmoid Activation Function in Neural Network

Question

I'm new into this and I'm trying to replace the sigmoid activation function in the following simple NN with ReLU. Can I do that? I've tried replacing the sigmoid function, but it's not working. The output should be the AND gate(if input (0,0)-> output 0).

import numpy as np

 # sigmoid function
def nonlin(x, deriv=False):
   if(deriv == True):
       return x*(1-x)
   return 1/(1+np.exp(-x))

# input dataset
X = np.array([[0, 0],
          [0, 1],
          [1, 0],
          [1, 1]])

# output dataset            
y = np.array([[0, 0, 0, 1]]).T

# seed random numbers to make calculation
# deterministic (just a good practice)
np.random.seed(1)

# initialize weights randomly with mean 0
syn0 = 2*np.random.random((2, 1)) - 1

for iter in xrange(10000):
    # forward propagation
    l0 = X
    l1 = nonlin(np.dot(l0,syn0))

    # how much did we miss?
    l1_error = y - l1

    l1_delta = l1_error * nonlin(l1, True)

    syn0 += np.dot(l0.T,l1_delta)

Answer 1

To answer your question directly, yes you can replace the sigmoid activation function with the ReLU activation function.

I think the real question is "should you?" This is a much harder question to answer.

The sigmoid function returns a number between 0 and 1, but the path of the function is never constant, decelerating as it approaches 0 and 1, and peaking at 0.5 when the input value is 0.

On the other hand, the ReLU function returns a 0 for input values less than 0, while input values above 0, the function returns a value between 0 and 1.

So, sigmoid will only "theoretically" return a 0 at the lowest most extreme value of the input $x$. For the mean input value of $x$, the sigmoid returns 0.5, and for the max value of $x$ will "theoretically" return a value of 1. Based on a normally distributed variable, you would not expect to see a 0 or 1 very frequently.

With the ReLU function, there is nothing theoretical about the possibility of the function returning a 0. It will return a 0 for approximately half the input values.

One other point, the activation function can absolutely change the behavior of a neural network, but a poorly chosen activation function may not necessarily prevent a NN from converging, but it will probably require a different model architecture, and probably make it harder to train.

At this point, the community appears to rely on heuristics and empirical assessment to choose one function over another. There are certain patterns that seem to be accepted by the community as a general rule of thumb in certain domains. There are also, some patterns that are pretty well accepted as the only valid pattern. For example, when using the softmax activation function to determine the output, it is generally accepted that the previous hidden layer must employ the sigmoid function.

My guess is that most practitioners start with a sigmoid activation function unless there is domain knowledge that implies that it would be inappropriate, but I expect many different architectures are evaluated during model selection.