To answer your question directly, yes you can replace the sigmoid
activation function with the ReLU
activation function.
I think the real question is "should you?" This is a much harder question to answer.
The sigmoid
function returns a number between 0 and 1, but the path of the function is never constant, decelerating as it approaches 0 and 1, and peaking at 0.5 when the input value is 0.
On the other hand, the ReLU
function returns a 0 for input values less than 0, while input values above 0, the function returns a value between 0 and 1.
So, sigmoid
will only "theoretically" return a 0 at the lowest most extreme value of the input $x$. For the mean input value of $x$, the sigmoid returns 0.5, and for the max value of $x$ will "theoretically" return a value of 1. Based on a normally distributed variable, you would not expect to see a 0 or 1 very frequently.
With the ReLU
function, there is nothing theoretical about the possibility of the function returning a 0. It will return a 0 for approximately half the input values.
One other point, the activation function can absolutely change the behavior of a neural network, but a poorly chosen activation function may not necessarily prevent a NN from converging, but it will probably require a different model architecture, and probably make it harder to train.
At this point, the community appears to rely on heuristics and empirical assessment to choose one function over another. There are certain patterns that seem to be accepted by the community as a general rule of thumb in certain domains. There are also, some patterns that are pretty well accepted as the only valid pattern. For example, when using the softmax
activation function to determine the output, it is generally accepted that the previous hidden layer must employ the sigmoid
function.
My guess is that most practitioners start with a sigmoid
activation function unless there is domain knowledge that implies that it would be inappropriate, but I expect many different architectures are evaluated during model selection.