Similar to this post, I need the gradient descent step of tanh but now with binary cross entropy (BCE).

So we have

$$ \Delta \omega = -\eta \frac{\delta E}{\delta \omega} $$

Now we have BCE:

$$ E = −(ylog(\hat{y})+(1−y)log(1−\hat{y})) $$

Considering my output is $\hat{y} = tanh(\omega .x)$, $x$ is my input vector and $y_i$ is the corresponding label here. $$ \frac{\delta E}{\delta \omega} = \frac{\delta −(ylog(tanh(wx))+(1−y)log(1−tanh(wx)))}{\delta \omega} $$

Now on this website they do something similar for the normal sigmoid and arrive at (eq 60):

$$ \frac{σ′(z)x}{ σ(z)(1−σ(z))}(σ(z)−y) $$

Could we use that and continue there? We can get the derivative like this and get:

$$ \frac{tanh′(wx)x}{tanh(wx)(1−tanh(wx))}(tanh(wx)−y) \\= \frac{x-xtanh(wx)^2}{tanh(wx)(1−tanh(wx))}(tanh(wx)−y) \\= \frac{x-x\hat{y}^2}{\hat{y}(1−\hat{y})}(\hat{y}−y) \\= \frac{(\hat{y} + 1)x(\hat{y} - y)}{\hat{y}} $$

Wherever I look, I don't find this :)


Given the first answer that gives $(1 + \hat{y})(1 - \hat{y})$, we arrive at the same

$$ \frac{tanh′(wx)x}{tanh(wx)(1−tanh(wx))}(tanh(wx)−y) \\= \frac{x(1 + \hat{y})(1 - \hat{y})}{\hat{y}(1−\hat{y})}(\hat{y}−y) \\= \frac{(\hat{y} + 1)x(\hat{y} - y)}{\hat{y}} $$


1 Answer 1


Let 'a' be the output from an activation function like sigmoid or tanh.

Therefore, the derivative of sigmoid is a*(1-a) whereas the derivative for tanh is (1+a)*(1-a).

Just follow the derivation of sigmoid except replace the derivative of sigmoid with that of tanh.

  • $\begingroup$ So that gives the same outcome. But isn't that problematic when $\hat{y} = 0$ (division by 0)? $\endgroup$
    – Roelant
    Apr 30, 2020 at 11:57
  • $\begingroup$ Since you have defined your loss BCE function which can take ground truth values (i.e., $y$) 1 or 0, you would need to use sigmoid function as your activation function to bound your output $\hat{y}$ between 0 and 1. If you still want to use tanh then look at the loss you concur when your $\hat{y} = 0$: it blows to infinity! Therefore using tanh as your activation for the defined form of BCE wouldn't make sense. $\endgroup$ Apr 30, 2020 at 20:47
  • $\begingroup$ @Roelant Do you still have some doubt? $\endgroup$ May 2, 2020 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.