Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity.

Why is ReLU used as an activation function?

Why ReLU is better than the other activation functions

Am I better off using a smoother activation like tanh if I have a smaller, dense dataset?

enter image description here


1 Answer 1


There is no reason to to predict tanh would be better if for a smaller, dense dataset.

The effect of activation function varies from dataset to dataset. You could it treat like an empirical question - find the best activation function for a specific data set like any other hyperparameter.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.