0
$\begingroup$

Background

It is said Sigmoid/Tanh are not to use because of the saturation issue.

enter image description here

Question

The saturation occurs when the input to Sigmoid/Tanh is a large value. Will normalizing the input between -1 and 1, for instance using Batch Normalization (BN), avoid the issue?

Please help understand if saturation of sigmoid/tanh is still the issue when such normalization is used.

enter image description here

If the issue is avoided, then is tanh still a valid activation which can avoid the dead ReLU issue?

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.