3
$\begingroup$

I have trained a model with linear activation function for the last dense layer, but I have a constraint that forbids negative values for the target which is a continuous positive value.

Can I use ReLU as the activation of the output layer? I am afraid of trying, since it is generally used in hidden layers as a rectifier. I'm using Keras.

$\endgroup$
3
$\begingroup$

Yes, you can. Basically, for regression tasks, it is customary to use the linear function as the non-linearity due to the fact that it's differentiable and it does not limit the output. This means you can make any output using your inputs. People do not use tanh or sigmoid as the activation function of the last layers for regression tasks due to the fact that they are limited and cannot generate all numbers which are needed. In your task, you can use ReLU as the non-linearity. The concept of non-linearities in hidden layers is to add non-linear boundaries and for the last layer in regression tasks, it should make all possible choices. In your case, ReLU is the best.

|improve this answer|||||
$\endgroup$
  • 1
    $\begingroup$ wonderful, even validation loss is getting lower. I hope it will work well on a ton of unseen data. $\endgroup$ – bacloud14 Mar 15 '19 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.