1
$\begingroup$

What is the purpose of linear activation functions in keras, isn't the entire point of activation functions to introduce non-linearity?

$\endgroup$
  • 1
    $\begingroup$ I'm not 100% sure, but the Activation ("linear") might be an historic artifact. There is / was also an activation parameter to a couple of functions. There it makes sense, because sometimes you don't want to apply an activation function. $\endgroup$ – Martin Thoma Nov 29 '18 at 7:03
1
$\begingroup$

A combination of multiple linear functions can help one model complex decision regions. Two or more linear functions can be combined to form a piece-wise approximation of a non linear function. Computation of non-linear activation function on a very deep network can be expensive (because of the exponential term involved). ReLu activation function which is a linear activation is used almost by every CNN model and is proved to give better results. $$ f(x) = max(0,x-1) + max(0,x+1)$$ Consider the function above which is combination of two reLu's. This approximates the sigmoid function(try plotting the same) and is computationally cheap.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.