To answer your question directly, yes you can replace the
sigmoid activation function with the
ReLU activation function.
I think the real question is "should you?" This is a much harder question to answer.
sigmoid function returns a number between 0 and 1, but the path of the function is never constant, decelerating as it approaches 0 and 1, and peaking at 0.5 when the input value is 0.
On the other hand, the
ReLU function returns a 0 for input values less than 0, while input values above 0, the function returns a value between 0 and 1.
sigmoid will only "theoretically" return a 0 at the lowest most extreme value of the input $x$. For the mean input value of $x$, the sigmoid returns 0.5, and for the max value of $x$ will "theoretically" return a value of 1. Based on a normally distributed variable, you would not expect to see a 0 or 1 very frequently.
ReLU function, there is nothing theoretical about the possibility of the function returning a 0. It will return a 0 for approximately half the input values.
One other point, the activation function can absolutely change the behavior of a neural network, but a poorly chosen activation function may not necessarily prevent a NN from converging, but it will probably require a different model architecture, and probably make it harder to train.
At this point, the community appears to rely on heuristics and empirical assessment to choose one function over another. There are certain patterns that seem to be accepted by the community as a general rule of thumb in certain domains. There are also, some patterns that are pretty well accepted as the only valid pattern. For example, when using the
softmax activation function to determine the output, it is generally accepted that the previous hidden layer must employ the
My guess is that most practitioners start with a
sigmoid activation function unless there is domain knowledge that implies that it would be inappropriate, but I expect many different architectures are evaluated during model selection.