Building on the questions below, the only conclusion I could draw from the answers was that ReLu
is less computationally expensive and better at sparsity.
Am I better off using a smoother activation like tanh
if I have a smaller, dense dataset?
Building on the questions below, the only conclusion I could draw from the answers was that ReLu
is less computationally expensive and better at sparsity.
Am I better off using a smoother activation like tanh
if I have a smaller, dense dataset?
There is no reason to to predict tanh would be better if for a smaller, dense dataset.
The effect of activation function varies from dataset to dataset. You could it treat like an empirical question - find the best activation function for a specific data set like any other hyperparameter.