# How to implement gradient descent for a tanh() activation function for a single layer perceptron?

I am required to implement a simple perceptron based neural network for an image classification task, with a binary output and a single layer, however I am having difficulties. I have a few problems:

1. I am required to use a tanh() axtivation function, which has the range [-1, 1], however, my training labels are 1 and 0. Should I scale the activation function, or simply change 0 labels to -1?

2. So the gradient descent rule dictates that I shift the weights according to the rule:

$$\Delta \omega = -\eta \frac{\delta E}{\delta \omega}$$

I am using the mean square error for my error:

$$E = (output - label)^2$$

Considering my output is $o = tanh(\omega .x)$, x is my input vector and $y_i$ is the corresponding label here. $$\frac{\delta E}{\delta \omega} = \frac{\delta (y_i - tanh(wx))^2}{\delta \omega} \\= -2(y_i - tanh(wx))(1 -tanh^2(wx)) \frac{\delta wx}{\delta w} \\= -2(y_i - tanh(wx))(1 -tanh^2(wx)) x \\= -2(y_i - o)(1 - o^2)x$$

I implemented this is python, the dot product of the input vector with the weights turns out to be too large, which makes $tanh(x)=1$ and $1-o^2 = 0$, so I can't learn. How can I circumvent this problem?

Thanks for the replies.

The implementation:

def perc_nnet(X, y, iter = 10000, eta=0.0001):
a, b, c = X.shape
W_aug = np.random.normal(0, 0.01, a*b+1)
errors = []

for i in range(iter):
selector = rd.randint(0,c-1)
x_n = X[:,:,selector].ravel() #.append(1) #has the bias as well
x_n = np.append(x_n, 1)
v = x_n.dot(W_aug)
o = np.tanh(v)
y_i = y[:,selector] if y[:,selector]==1 else -1
MSE = 0.5*(o - y_i)**2
errors.append(MSE)
delta = - eta * (o - y_i) * (1 - o**2) * x_n
W_aug = W_aug + delta

return W_aug, errors

• Can you provide your Python implementation? – E_net4 posts memes Oct 17 '17 at 11:50
• I updated the question with the code – Waylander Oct 17 '17 at 13:05

How can I circumvent this problem?

TLDR : Normalize your input data.

Why?

Notice how tanh actually works on the input data:

>>> np.tanh(np.asarray([-1000, -100, -10, -1, 0, 1, 10, 100, 1000])) array([-1. , -1., -1. , -0.76159416, 0.,0.76159416, 1. ,1. , 1])

If your data input to tanh has a magnitude of 10 or more, tanh produces an output of 1. That means it treats 10, 50, 100, 1000 the same. We don't want that.

This would explain

the input vector with the weights turns out to be too large, which makes tanh(x)=1tanh(x)=1 and 1−o2=01−o2=0, so I can't learn.

Instead, we normalize the data (by dividing the input data by 1000 ( there are other ways of doing this)):

>>> np.tanh(np.asarray([-1000, -100, -10, -1, 0, 1, 10, 100, 1000]) / 1000) array([-0.76159416, -0.09966799, -0.00999967, -0.001, 0. ,0.001, 0.00999967, 0.09966799, 0.76159416])

Now, tanh treats all of them differently.

Note You are doing a classification task. Using mean squared error as a Cost Function won't yield the best results here. You should instead use Cross Entropy Loss with One Hot Vectors and softmax.

As to why should be a whole different answer. Here is a link for the same.

• Ah, seems so obvious now, a rookie mistake. Thanks. – Waylander Oct 18 '17 at 8:29