5
$\begingroup$

I am trying to build a deep neural network that learns the coordinate-coordinate bitwise XOR of two matrices, but it performs poorly.

For example, in the 2 bits case, its accuracy stays around 0.5. Here is the code snippet:

from keras.layers import Dense, Activation
from keras.layers import Input
import numpy as np 
from keras.layers.merge import concatenate
from keras.models import Model


size=1
data1 = np.random.choice([0, 1], size=(50000,size,size))
data2 = np.random.choice([0, 1], size=(50000,size,size))
labels  = np.bitwise_xor(data1, data2)
a = Input(shape=(size,size))
b = Input(shape=(size,size))
a1 = Dense(size, activation='sigmoid')(a)
b1 = Dense(size, activation='sigmoid')(b)
merged = concatenate([a1, b1])
hidden = Dense(1, activation='sigmoid')(merged)
hidden = Dense(3, activation='sigmoid')(hidden)
hidden = Dense(5, activation='relu')(hidden)
hidden = Dense(4, activation='sigmoid')(hidden)
hidden = Dense(3, activation='sigmoid')(hidden)
outputs = Dense(1, activation='relu')(hidden)

model = Model(inputs=[a, b], outputs=outputs)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit([data1, data2], np.array(labels), epochs=15, batch_size=32)

What's going on here?

Epoch 1/15
50000/50000 [==============================] - 7s 130us/step - loss: 0.7118 - acc: 0.5044
Epoch 2/15
50000/50000 [==============================] - 4s 78us/step - loss: 0.6933 - acc: 0.5023
Epoch 3/15
50000/50000 [==============================] - 4s 74us/step - loss: 0.6934 - acc: 0.5030
Epoch 4/15
50000/50000 [==============================] - 4s 86us/step - loss: 0.6935 - acc: 0.5002
Epoch 5/15
50000/50000 [==============================] - 4s 79us/step - loss: 0.6934 - acc: 0.5015
Epoch 6/15
50000/50000 [==============================] - 5s 96us/step - loss: 0.6935 - acc: 0.5030
Epoch 7/15
50000/50000 [==============================] - 5s 105us/step - loss: 0.6934 - acc: 0.5026
$\endgroup$
4
$\begingroup$

I think there might be a few things going on.

You might have a reason but I don't know why you have shaped your input data into three dimensions: size=(50000,size,size).

Also, you might have a reason but I don't know why you ran each feature separately through a different layer (each with a single hidden unit), and then merged the outputs before running the merged output through another series of layers:

a = Input(shape=(size,size))
b = Input(shape=(size,size))
a1 = Dense(size, activation='sigmoid')(a)
b1 = Dense(size, activation='sigmoid')(b)
merged = concatenate([a1, b1])

Also, I suspect that running the features through a single hidden unit reduces the information sent through the rest of the network, so the network cannot learn the XOR function.

Here is some code that works for me:

from keras import models

from keras.layers import Dense

import numpy as np

Simulate data:

X_1 = np.random.choice([0, 1], size = (50000, 1))
X_2 = np.random.choice([0, 1], size = (50000, 1))

X = np.concatenate((X_1, X_2), axis = 1)

Y = np.bitwise_xor(X[:, 0], X[:, 1])

FNN Model:

# Define model.

network_fnn = models.Sequential()
network_fnn.add(Dense(4, activation = 'relu', input_shape = (X.shape[1],)))
network_fnn.add(Dense(4, activation = 'relu'))
network_fnn.add(Dense(1, activation = 'sigmoid'))

# Compile model.

network_fnn.compile(optimizer = 'rmsprop', loss = 'binary_crossentropy', metrics = ['acc'])

# Fit model.

history_fnn = network_fnn.fit(X, Y, epochs = 5, batch_size = 32, verbose = True)
$\endgroup$
  • 1
    $\begingroup$ You said: You might have a reason but I don't know why you have shaped your input data into three dimensions: size=(50000,size,size). I wanted to create a data set of 50000 examples to train the network. $\endgroup$ – 0x90 Sep 5 '18 at 8:34
  • 1
    $\begingroup$ Of course. But why did you create 3 dimensions [50000, 1, 1]? $\endgroup$ – from keras import michael Sep 5 '18 at 20:08
  • 1
    $\begingroup$ Not sure I fully understand, I wanted to create 50,000 examples of matrices of size 1x1. (Later I will take the code to NxN matrices.) $\endgroup$ – 0x90 Sep 5 '18 at 20:21
  • $\begingroup$ I see. In the XOR case, the extra dimensions seemed superfluous. Seems like you have a plan, though! look = np.random.choice([0, 1], size = (5, 1, 1)) look look.shape look = np.random.choice([0, 1], size = (5, 1)) look look.shape look = np.random.choice([0, 1], size = (5)) look look.shape $\endgroup$ – from keras import michael Sep 6 '18 at 1:50
3
$\begingroup$

The real reason is that you are using activation='relu' in the output layer. For binary classification you must use sigmoid.

Then you chose a poor architecture, I would suggest:

a = Input(shape=(size,size))
b = Input(shape=(size,size))
a1 = Dense(size, activation='sigmoid')(a)
b1 = Dense(size, activation='sigmoid')(b)
merged = concatenate([a1, b1])
hidden = Dense(5, activation='relu')(merged)
hidden = Dense(5, activation='relu')(hidden)
hidden = Dense(5, activation='relu')(hidden)

outputs = Dense(1, activation='sigmoid')(hidden)
model = Model(inputs=[a, b], outputs=outputs)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit([data1, data2], np.array(labels), epochs=15, batch_size=32)
$\endgroup$
  • $\begingroup$ Why the architecture I chose is poor? $\endgroup$ – 0x90 Sep 29 '18 at 22:52
  • 2
    $\begingroup$ Good question. I guess you will learn with experience to decide architectures. For example, the first layer with just one unit is really uncommon, you are likely to lose the useful information from the input. Then you used too many layers, so the network is to deep for this task. And I don't like the rising number of units from the input to the middle. $\endgroup$ – Francesco Pegoraro Sep 29 '18 at 22:59
  • 1
    $\begingroup$ I agree. The original model architecture is far too complex for the XOR problem. Even in my example code above, you could get rid of the second hidden layer if you increase the number of units in the first hidden layer. Rather than starting with a complex architecture, one is better off to start with a simple architecture, and then increase the complexity if needed. $\endgroup$ – from keras import michael Sep 30 '18 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.