How to implement gradient descent for a tanh() activation function for a single layer perceptron?

Question

I am required to implement a simple perceptron based neural network for an image classification task, with a binary output and a single layer, however I am having difficulties. I have a few problems:

1. I am required to use a tanh() axtivation function, which has the range [-1, 1], however, my training labels are 1 and 0. Should I scale the activation function, or simply change 0 labels to -1?

2. So the gradient descent rule dictates that I shift the weights according to the rule:

$$ \Delta \omega = -\eta \frac{\delta E}{\delta \omega} $$

I am using the mean square error for my error:

$$ E = (output - label)^2 $$

Considering my output is $o = tanh(\omega .x)$, x is my input vector and $y_i$ is the corresponding label here. $$ \frac{\delta E}{\delta \omega} = \frac{\delta (y_i - tanh(wx))^2}{\delta \omega} \\= -2(y_i - tanh(wx))(1 -tanh^2(wx)) \frac{\delta wx}{\delta w} \\= -2(y_i - tanh(wx))(1 -tanh^2(wx)) x \\= -2(y_i - o)(1 - o^2)x $$

I implemented this is python, the dot product of the input vector with the weights turns out to be too large, which makes $tanh(x)=1$ and $1-o^2 = 0$, so I can't learn. How can I circumvent this problem?

Thanks for the replies.

The implementation:

def perc_nnet(X, y, iter = 10000, eta=0.0001):
    a, b, c = X.shape
    W_aug = np.random.normal(0, 0.01, a*b+1)
    errors = []

    for i in range(iter):
        selector = rd.randint(0,c-1)
        x_n = X[:,:,selector].ravel() #.append(1) #has the bias as well
        x_n = np.append(x_n, 1)
        v = x_n.dot(W_aug)
        o = np.tanh(v)
        y_i = y[:,selector] if y[:,selector]==1 else -1
        MSE = 0.5*(o - y_i)**2
        errors.append(MSE)
        delta = - eta * (o - y_i) * (1 - o**2) * x_n
        W_aug = W_aug + delta

    return W_aug, errors

Answer 1

How can I circumvent this problem?

TLDR : Normalize your input data.

Why?

Notice how tanh actually works on the input data:

>>> np.tanh(np.asarray([-1000, -100, -10, -1, 0, 1, 10, 100, 1000])) array([-1. , -1., -1. , -0.76159416, 0.,0.76159416, 1. ,1. , 1])

If your data input to tanh has a magnitude of 10 or more, tanh produces an output of 1. That means it treats 10, 50, 100, 1000 the same. We don't want that.

This would explain

the input vector with the weights turns out to be too large, which makes tanh(x)=1tanh(x)=1 and 1−o2=01−o2=0, so I can't learn.

Instead, we normalize the data (by dividing the input data by 1000 ( there are other ways of doing this)):

>>> np.tanh(np.asarray([-1000, -100, -10, -1, 0, 1, 10, 100, 1000]) / 1000) array([-0.76159416, -0.09966799, -0.00999967, -0.001, 0. ,0.001, 0.00999967, 0.09966799, 0.76159416])

Now, tanh treats all of them differently.

Note You are doing a classification task. Using mean squared error as a Cost Function won't yield the best results here. You should instead use Cross Entropy Loss with One Hot Vectors and softmax.

As to why should be a whole different answer. Here is a link for the same.