My neural network is made only by two hidden fully connected units. I've obtained very good results using relu as the activation function, and only good results using softplus.
My main purpose is to obtain the gradient of the output w.r.t. the input as accurate as possible. Since relu has a step-like derivative, it's better using softplus as my activation function even if I got better predictions with relu?
Thanks for your time, Alberto