The one of the benefits of ReLUs is sparsity. Sparsity arises when a≤0 (a = wX+b). The more such units that exist in a layer the more sparse the resulting representation. Sigmoids on the other hand are always likely to generate some non-zero value resulting in dense representations. Sparse representations seem to be more beneficial than dense representations. Is it true that sparse representation is more beneficial that dense representation, especially for Neural Networks?
Sparse representations are not always better than Dense, though they do have an upper hand. In terms of various inputs being handled (varying amount of information), without a sparse prior we shall be trying to learn same size fit for all representation, that may be inadequate. Here dense representations will have lesser degrees of freedom, resulting in non-linear relationship. Overall our representation mathematically needs to mimic the topology of underlying manifold. But in how many scenarios can we have sparse representation, is the bigger question/problem, where we we have sufficient uncorrupted identities. Approaches like SLR dictionary decomposition come in handy in such scenarios.