I am currently preparing for an exam in neural networks. In several protocols from former exams I read that activation functions of neurons (in multilayer perceptrons) have to be monotonic.

I understand that activation functions should be differentiable, have a derivative which is not 0 on most points and non-linear. I do not undertand why being monotonic is important / helpful.

I know the following activation functions and that they are monotonic:

  • ReLU
  • Sigmoid
  • Tanh
  • Softmax: I'm not sure if the definition of monotonicity is applicable for functions $f: \mathbb{R}^n \rightarrow \mathbb{R}^m$ with $n, m > 1$
  • Softplus
  • (Identity)

However, I still can't see any reason why for example $\varphi(x) = x^2$.

(Why) do activation functions have to be monotonic?

(Related side question: Is there any reason why the logarithm / exponential function is not used as an activation function?)

  • 3
  • 1
    @MartinThoma Are you sure softmax is monotonic? – Media Feb 21 at 7:07
  • 1
    Thanks @Media. To answer your question: I'm not sure what "monotonic" even means for functions in $f:R^n \rightarrow R^m$ with $m > 1$. For $m=1$ softmax is constant and thus monotonic. But without defining $<$ for elements in $R^n$ with $n>1$ I don't think monotonic makes any sense. – Martin Thoma Feb 21 at 19:50
  • 1
    @MartinThoma Thanks, actually it was also a question of mine. I didn't know, and still don't know, if there is an extension for monotonic in functions with multiple outputs. Math stuff, you know! – Media Feb 22 at 14:06
up vote 10 down vote accepted

The monotonicity criterion helps the neural network to converge easier into an more accurate classifier. See this stackexchange answer and wikipedia article for further details and reasons.

However, the monotonicity criterion is not mandatory for an activation function - It is also possible to train neural nets with non-monotonic activation functions. It just gets harder to optimize the neural network. See Yoshua Bengio's answer.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.