0
$\begingroup$

Can we use a modulo function $f(x)$ as activation function in a neural network? Modulo function is monotonic and continuous (just like Relu) except at a finite number of points in the domain of our input data. By modulo function $f(x)$ I mean

\begin{equation} f(x) = \begin{bmatrix} \vdots \\ x+a \ \ \ \if \ -a < x < 0 \\ x \ \ \ \ \ \ \ \if\ \ \ \ 0 < x < a\\ x-a \ \ \ \if \ \ \ \ a < x < 2a \\ \vdots \\ \end{bmatrix} \end{equation} where a is a positive constant number and could be treated as hyperparameter for simplicity.

I want my output to take values between [0,1] and I am sampling the output from a gaussian distribution 𝑁(𝜇, $\sigma^2$) where 𝜇, $\sigma^2$ are the outputs of neural network. Hence the output may go outside the range [0,1]. I don't want to do clipping because it will create further problems in my network

I am new to Latex, sorry for not using a good formating.

$\endgroup$
  • $\begingroup$ Hue you are defining the Relationship between X and A.? And also what the goal that the available activation functions arent able to solve? $\endgroup$ – abheet22 Oct 5 '19 at 5:37
  • $\begingroup$ I want my output to take values between [0,1] and I am sampling the output from a gaussian distribution $N(\mu,\sigma^2)$ where $\mu$, $\sigma^2$ are the outputs of neural network. Hence the output may go outside the range [0,1]. I don't want to do clipping because it will create further problems in my network. $\endgroup$ – Japneet Singh Oct 6 '19 at 4:51
1
$\begingroup$

The modulo function is not monotonic. Even though it has a positive derivative except at points where it is not differentiable. It is generally not recommended to use a non-convex Activation function. Hence, it is better to avoid it or change the structure of problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.