1
$\begingroup$

Is there a closed form solution for a ReLU that is bounded with a maximum value at 1? I am trying to produce output values for pixel intensities 0 <= x <= 1, but my outputs are producing values greater than 1. How can I counteract this?

$\endgroup$
3
  • $\begingroup$ Are you using a specific framework? Many will allow you to define your own transfer functions (e.g. this should be straightforward to add in TensorFlow) $\endgroup$ Nov 23 '16 at 19:47
  • 1
    $\begingroup$ "Is there a closed form solution for a ReLU that is bounded with a maximum value at 1?" - closed form solution of what? $\endgroup$ Nov 25 '16 at 15:37
  • $\begingroup$ "I am trying to produce output values for pixel intensities 0 <= x <= 1" - why do you want this? $\endgroup$ Nov 25 '16 at 15:37
4
$\begingroup$

Being itself a non-linearity, the very advantage of ReLUs is their linear regime, that prevents it from saturating too early.

Having said that, a couple of alternatives for your case are to either clip the output of the ReLUs or to place sigmoid (or tanh, adjusting the output range) after them (or instead of them).

$\endgroup$
0
$\begingroup$

Use a sigmoid or tanh. I think it's pointless to use a clipped ReLU in your case also because, in addition to what @ncasas said, for values less than zero or greater than your clipping threshold (i.e. 1) you would not get a gradient, possibly making learning harder.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.