Are non-relu activations better for small/ dense datasets?

Question

Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity.

Why is ReLU used as an activation function?

Why ReLU is better than the other activation functions

Am I better off using a smoother activation like tanh if I have a smaller, dense dataset?

Answer 1

There is no reason to to predict tanh would be better if for a smaller, dense dataset.

The effect of activation function varies from dataset to dataset. You could it treat like an empirical question - find the best activation function for a specific data set like any other hyperparameter.