# What is the gradient descent rule using binary cross entropy (BCE) with tanh?

Similar to this post, I need the gradient descent step of tanh but now with binary cross entropy (BCE).

So we have

$$\Delta \omega = -\eta \frac{\delta E}{\delta \omega}$$

Now we have BCE:

$$E = −(ylog(\hat{y})+(1−y)log(1−\hat{y}))$$

Considering my output is $$\hat{y} = tanh(\omega .x)$$, $$x$$ is my input vector and $$y_i$$ is the corresponding label here. $$\frac{\delta E}{\delta \omega} = \frac{\delta −(ylog(tanh(wx))+(1−y)log(1−tanh(wx)))}{\delta \omega}$$

Now on this website they do something similar for the normal sigmoid and arrive at (eq 60):

$$\frac{σ′(z)x}{ σ(z)(1−σ(z))}(σ(z)−y)$$

Could we use that and continue there? We can get the derivative like this and get:

$$\frac{tanh′(wx)x}{tanh(wx)(1−tanh(wx))}(tanh(wx)−y) \\= \frac{x-xtanh(wx)^2}{tanh(wx)(1−tanh(wx))}(tanh(wx)−y) \\= \frac{x-x\hat{y}^2}{\hat{y}(1−\hat{y})}(\hat{y}−y) \\= \frac{(\hat{y} + 1)x(\hat{y} - y)}{\hat{y}}$$

Wherever I look, I don't find this :)

Update

Given the first answer that gives $$(1 + \hat{y})(1 - \hat{y})$$, we arrive at the same

$$\frac{tanh′(wx)x}{tanh(wx)(1−tanh(wx))}(tanh(wx)−y) \\= \frac{x(1 + \hat{y})(1 - \hat{y})}{\hat{y}(1−\hat{y})}(\hat{y}−y) \\= \frac{(\hat{y} + 1)x(\hat{y} - y)}{\hat{y}}$$

• So that gives the same outcome. But isn't that problematic when $\hat{y} = 0$ (division by 0)? – Roelant Apr 30 '20 at 11:57
• Since you have defined your loss BCE function which can take ground truth values (i.e., $y$) 1 or 0, you would need to use sigmoid function as your activation function to bound your output $\hat{y}$ between 0 and 1. If you still want to use tanh then look at the loss you concur when your $\hat{y} = 0$: it blows to infinity! Therefore using tanh as your activation for the defined form of BCE wouldn't make sense. – user1825567 Apr 30 '20 at 20:47