0
$\begingroup$

I've created a simple 2-2-1 feedforward ANN to predict an XOR using Keras.

The activation function I'm using on all layers is a tanh, so in order to make use of the entire range of the function, i.e. [-1, 1], I've decided to use -1 instead of 0 as the symbol.

My input data is, thus, [[-1, -1], [-1, 1], [1, -1], [1, 1]], for an output of [[-1], [1], [1], [-1]].

I thought this would give better results since I'm using the entire range of the function and it supposedly would converge better because of that. Also, since I'm just using a different symbol, it should be the same as with using 0 and 1.

However, my network can not converge (giving a 0.5 accuracy), and what baffles me the most is that using 0 and 1 as symbols converges, and at a much faster rate.

Is there a reason for such a counter-intuitive (at least in my conception) thing to be happening?

$\endgroup$

1 Answer 1

1
$\begingroup$

I tried this experiment and was able to get some positive results. I will describe what I tried then perhaps you can specify where the differences may lie and we can further explore them. From what I tried I would assume that you are simply not training long enough.

Creating the data

import numpy as np

n = 100000
x_train = np.zeros((n,2))
y_train = np.zeros((n,))
for i in range(n):
    x_train[i,0] = np.random.choice([-1,1])
    x_train[i,1] = np.random.choice([-1,1])
    if x_train[i,0] == 1 and x_train[i,1] == 1 or x_train[i,0] == -1 and x_train[i,1] == -1:
        y_train[i] = -1
    else:
        y_train[i] = 1

x_train = x_train.reshape(n, 2,)

n = 1000
x_test = np.zeros((n,2))
y_test = np.zeros((n,))
for i in range(n):
    x_test[i,0] = np.random.choice([-1,1])
    x_test[i,1] = np.random.choice([-1,1])
    if x_test[i,0] == 1 and x_test[i,1] == 1 or x_test[i,0] == -1 and x_test[i,1] == -1:
        y_test[i] = -1
    else:
        y_test[i] = 1

x_test = x_test.reshape(n, 2,)

print(x_test[0].T) 
print(y_test[0])

[ 1. 1.]
-1.0

Build the model

As you describe the model is 2 input nodes, 2 hidden nodes and 1 output node. Every node is using tanh as its activation function.

input_shape = (2,)

model = Sequential()
model.add(Dense(2, activation='tanh',
                 input_shape=input_shape))
model.add(Dense(2, activation='tanh'))
model.add(Dense(1, activation='tanh'))

model.compile(loss=keras.losses.mean_squared_error,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])

Train the model

Because I generated many instances of the data I am only training for 10 epochs. However, if your input space is only the four possible inputs you may want thousands of epochs. Neural networks do take a long time to converge.

epochs = 10
batch_size = 128
# Fit the model weights.
history = model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
          verbose=1,
          validation_data=(x_test, y_test))

Epoch 10/10
100000/100000 [==============================] - 1s 9us/step - loss: 9.3983e-05 - acc: 1.0000 - val_loss: 7.9096e-05 - val_acc: 1.0000

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.