What is the purpose of linear activation functions in keras, isn't the entire point of activation functions to introduce non-linearity?
1
-
1$\begingroup$ I'm not 100% sure, but the Activation ("linear") might be an historic artifact. There is / was also an activation parameter to a couple of functions. There it makes sense, because sometimes you don't want to apply an activation function. $\endgroup$– Martin ThomaNov 29, 2018 at 7:03
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
A combination of multiple linear functions can help one model complex decision regions. Two or more linear functions can be combined to form a piece-wise approximation of a non linear function. Computation of non-linear activation function on a very deep network can be expensive (because of the exponential term involved). ReLu activation function which is a linear activation is used almost by every CNN model and is proved to give better results. $$ f(x) = max(0,x-1) + max(0,x+1)$$ Consider the function above which is combination of two reLu's. This approximates the sigmoid function(try plotting the same) and is computationally cheap.
