0
$\begingroup$

Can we use a modulo function $f(x)$ as activation function in a neural network? Modulo function is monotonic and continuous (just like Relu) except at a finite number of points in the domain of our input data. By modulo function $f(x)$ I mean

\begin{equation} f(x) = \begin{bmatrix} \vdots \\ x+a \ \ \ \if \ -a < x < 0 \\ x \ \ \ \ \ \ \ \if\ \ \ \ 0 < x < a\\ x-a \ \ \ \if \ \ \ \ a < x < 2a \\ \vdots \\ \end{bmatrix} \end{equation} where a is a positive constant number and could be treated as hyperparameter for simplicity.

I want my output to take values between [0,1] and I am sampling the output from a gaussian distribution ๐‘(๐œ‡, $\sigma^2$) where ๐œ‡, $\sigma^2$ are the outputs of neural network. Hence the output may go outside the range [0,1]. I don't want to do clipping because it will create further problems in my network

I am new to Latex, sorry for not using a good formating.

$\endgroup$
2
  • $\begingroup$ Hue you are defining the Relationship between X and A.? And also what the goal that the available activation functions arent able to solve? $\endgroup$ – abheet22 Oct 5 '19 at 5:37
  • $\begingroup$ I want my output to take values between [0,1] and I am sampling the output from a gaussian distribution $N(\mu,\sigma^2)$ where $\mu$, $\sigma^2$ are the outputs of neural network. Hence the output may go outside the range [0,1]. I don't want to do clipping because it will create further problems in my network. $\endgroup$ – Japneet Singh Oct 6 '19 at 4:51
1
$\begingroup$

The modulo function is not monotonic. Even though it has a positive derivative except at points where it is not differentiable. It is generally not recommended to use a non-convex Activation function. Hence, it is better to avoid it or change the structure of problem.

$\endgroup$

Your Answer

By clicking โ€œPost Your Answerโ€, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.