A Neural Network That Learns Bitwise XOR

I am trying to build a deep neural network that learns the coordinate-coordinate bitwise XOR of two matrices, but it performs poorly.

For example, in the 2 bits case, its accuracy stays around 0.5. Here is the code snippet:

from keras.layers import Dense, Activation
from keras.layers import Input
import numpy as np
from keras.layers.merge import concatenate
from keras.models import Model

size=1
data1 = np.random.choice([0, 1], size=(50000,size,size))
data2 = np.random.choice([0, 1], size=(50000,size,size))
labels  = np.bitwise_xor(data1, data2)
a = Input(shape=(size,size))
b = Input(shape=(size,size))
a1 = Dense(size, activation='sigmoid')(a)
b1 = Dense(size, activation='sigmoid')(b)
merged = concatenate([a1, b1])
hidden = Dense(1, activation='sigmoid')(merged)
hidden = Dense(3, activation='sigmoid')(hidden)
hidden = Dense(5, activation='relu')(hidden)
hidden = Dense(4, activation='sigmoid')(hidden)
hidden = Dense(3, activation='sigmoid')(hidden)
outputs = Dense(1, activation='relu')(hidden)

model = Model(inputs=[a, b], outputs=outputs)
model.fit([data1, data2], np.array(labels), epochs=15, batch_size=32)


What's going on here?

Epoch 1/15
50000/50000 [==============================] - 7s 130us/step - loss: 0.7118 - acc: 0.5044
Epoch 2/15
50000/50000 [==============================] - 4s 78us/step - loss: 0.6933 - acc: 0.5023
Epoch 3/15
50000/50000 [==============================] - 4s 74us/step - loss: 0.6934 - acc: 0.5030
Epoch 4/15
50000/50000 [==============================] - 4s 86us/step - loss: 0.6935 - acc: 0.5002
Epoch 5/15
50000/50000 [==============================] - 4s 79us/step - loss: 0.6934 - acc: 0.5015
Epoch 6/15
50000/50000 [==============================] - 5s 96us/step - loss: 0.6935 - acc: 0.5030
Epoch 7/15
50000/50000 [==============================] - 5s 105us/step - loss: 0.6934 - acc: 0.5026


I think there might be a few things going on.

You might have a reason but I don't know why you have shaped your input data into three dimensions: size=(50000,size,size).

Also, you might have a reason but I don't know why you ran each feature separately through a different layer (each with a single hidden unit), and then merged the outputs before running the merged output through another series of layers:

a = Input(shape=(size,size))
b = Input(shape=(size,size))
a1 = Dense(size, activation='sigmoid')(a)
b1 = Dense(size, activation='sigmoid')(b)
merged = concatenate([a1, b1])


Also, I suspect that running the features through a single hidden unit reduces the information sent through the rest of the network, so the network cannot learn the XOR function.

Here is some code that works for me:

from keras import models

from keras.layers import Dense

import numpy as np


Simulate data:

X_1 = np.random.choice([0, 1], size = (50000, 1))
X_2 = np.random.choice([0, 1], size = (50000, 1))

X = np.concatenate((X_1, X_2), axis = 1)

Y = np.bitwise_xor(X[:, 0], X[:, 1])


FNN Model:

# Define model.

network_fnn = models.Sequential()
network_fnn.add(Dense(4, activation = 'relu', input_shape = (X.shape[1],)))

# Compile model.

network_fnn.compile(optimizer = 'rmsprop', loss = 'binary_crossentropy', metrics = ['acc'])

# Fit model.

history_fnn = network_fnn.fit(X, Y, epochs = 5, batch_size = 32, verbose = True)

• You said: You might have a reason but I don't know why you have shaped your input data into three dimensions: size=(50000,size,size). I wanted to create a data set of 50000 examples to train the network. – 0x90 Sep 5 '18 at 8:34
• Of course. But why did you create 3 dimensions [50000, 1, 1]? – from keras import michael Sep 5 '18 at 20:08
• Not sure I fully understand, I wanted to create 50,000 examples of matrices of size 1x1. (Later I will take the code to NxN matrices.) – 0x90 Sep 5 '18 at 20:21
• I see. In the XOR case, the extra dimensions seemed superfluous. Seems like you have a plan, though! look = np.random.choice([0, 1], size = (5, 1, 1)) look look.shape look = np.random.choice([0, 1], size = (5, 1)) look look.shape look = np.random.choice([0, 1], size = (5)) look look.shape – from keras import michael Sep 6 '18 at 1:50

The real reason is that you are using activation='relu' in the output layer. For binary classification you must use sigmoid.

Then you chose a poor architecture, I would suggest:

a = Input(shape=(size,size))
b = Input(shape=(size,size))
a1 = Dense(size, activation='sigmoid')(a)
b1 = Dense(size, activation='sigmoid')(b)
merged = concatenate([a1, b1])
hidden = Dense(5, activation='relu')(merged)
hidden = Dense(5, activation='relu')(hidden)
hidden = Dense(5, activation='relu')(hidden)

outputs = Dense(1, activation='sigmoid')(hidden)
model = Model(inputs=[a, b], outputs=outputs)