As far as I understand (pardon me if I am wrong) the activation functions in a neural network go through the following transformations:
- Multiplication by constants(weights) to x ( $f(ax)$ , $f(x)$ being the activation function).
- Recursive substitution $f(f(x))$.
Now with the above transformations a ReLU activation function should never be able to fit a x² curve. It can approximate, but as the input grows the error of that approximated function will also grow exponentially, right?
Now x² is a simple curve. How can ReLU perform better for real data which will be way more complicated than x²?
I am new to machine learning. So please pardon me if there are any blunders in anything I am assuming.