Activation function vs If else statement

Question

The question is very naive and most of us may know the answer. I have googled it but was not able to find a satisfactory answer so posting it here. Can someone please put the right words on this question.

Activation functions like ReLU, Sigmoid etc generally work like if else statements. e.g. if value of input is less than 0 then 0 else same value ( max(0,a) in case of ReLU) etc. So the question is why if else statements are not being used? Are these functions less compute intensive or work better in case of multidimensional data?

The argument in favor of if else statements is, the programming of neural networks is also being done in high level languages like python, C etc. In that case what compels the use of these activation functions rather than programmatic oriented If Else syntax?

Answer 1

Activation functions in general cannot be implemented with if-else statements.

ReLU is a particularly simple activation function, and can thus easily be implemented with an if-else block. For example (in Python):

def relu(x):
    return x if x > 0 else 0

But this only works because ReLU is linear for $ x \geq 0$.

What about sigmoidal activation functions?

There's no if/else implementation for this function that's analogous to that of ReLU. In fact, this is a one-liner with no if/else involved:

def approximate_logistic(x):
    return 1 / (1 + 2.71828**-x)

Same goes for lots of other activation functions like Softplus, CELU, Tanh.

If/else blocks simply aren't powerful enough to capture the wide variety of activation functions that we might want to use.