My neural network is made only by two hidden fully connected units. I've obtained very good results using relu as the activation function, and only good results using softplus.

My main purpose is to obtain the gradient of the output w.r.t. the input as accurate as possible. Since relu has a step-like derivative, it's better using softplus as my activation function even if I got better predictions with relu?

Thanks for your time, Alberto


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.