3
$\begingroup$

As far as I understand (pardon me if I am wrong) the activation functions in a neural network go through the following transformations:

  1. Multiplication by constants(weights) to x ( $f(ax)$ , $f(x)$ being the activation function).
  2. Recursive substitution $f(f(x))$.

Now with the above transformations a ReLU activation function should never be able to fit a x² curve. It can approximate, but as the input grows the error of that approximated function will also grow exponentially, right?

Now x² is a simple curve. How can ReLU perform better for real data which will be way more complicated than x²?

I am new to machine learning. So please pardon me if there are any blunders in anything I am assuming.

$\endgroup$
1
$\begingroup$

It can approximate, but as the input grows the error of that approximated function will also grow exponentially, right?

You are right, the neural network uses ReLu functions to approximate the output $f(x) = x^2$ and there is some error incurred, but you are forgetting that

  1. Non-linear functions also do approximation since we use only some subset of activation functions and incur some error which can be as huge as with ReLus

  2. There is also a number of neurons to consider. The more neurons/layers you have - the better the approximation can be (since more small-interval ReLus are fit to better approach the shape of $x^2$). If your input size grows, you should also consider increasing the number of neurons/layers, in this way the approximation will always be close enough.

Now x² is a simple curve. How can ReLU perform better for real data which will be way more complicated than x²?

The main reason for this is that even though other activation functions can approximate $x^2$ or something else better, they are not optimised as fast. Generally speaking, non-linear activation function decision surface is more complex than the loss created by ReLus and can contain better global minimum but this minimum is harder to find, and, thus, with the less sophisticated loss created by ReLus, we can find a better optima using some gradient descent procedure, and it will be much faster.

$\endgroup$
7
  • $\begingroup$ Wouldn't we prefer accuracy to speed. i mean we only need to get to the approximation once, don't we. Also is speed and simplicity the only reason reLU is considered better. $\endgroup$ – Adithya Sama Aug 17 '18 at 12:43
  • $\begingroup$ Speed here generally means that it is 1 month for tanh and 1 hour for Relu :) So, you find some optimum of ReLu loss in 1 hour, which gives you say 10% off prediction, or you find 1% off using tanh in 1 month, and 10% off in 1 week. This difference is prohibitive. And secondly, your loss function with tanh is much more difficult so you get stuck at local optima more than with ReLus, which results in always better optimum, say 10%for ReLus vs 25% off with tanh. Time frames/optima are just for illustration - a hyperbola. $\endgroup$ – Artem Moskalev Aug 17 '18 at 12:58
  • $\begingroup$ so basically when you use reLU the number of nodes need to be high to match the complexity needed. and as its fast it will converge faster. $\endgroup$ – Adithya Sama Aug 17 '18 at 15:22
  • $\begingroup$ ReLu is used for the second reason - faster convergence, which due to limitation of computational resources/better behaving function results in better optima, which results in better performance metric results. As for the number of nodes - it works with any activation, not only ReLu. $\endgroup$ – Artem Moskalev Aug 17 '18 at 17:52
  • $\begingroup$ yeah i get the faster part. But what i mean is the final approximation accuracy that reLU can ever reach will always be less than what other (curvy) functions can achieve right?. When i mean always i mean when it comes to fitting curves like x^2 $\endgroup$ – Adithya Sama Aug 18 '18 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.