It is said Sigmoid/Tanh are not to use because of the saturation issue.

enter image description here


The saturation occurs when the input to Sigmoid/Tanh is a large value. Will normalizing the input between -1 and 1, for instance using Batch Normalization (BN), avoid the issue?

Please help understand if saturation of sigmoid/tanh is still the issue when such normalization is used.

enter image description here

If the issue is avoided, then is tanh still a valid activation which can avoid the dead ReLU issue?

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.