Xor gate accuracy improvement

Question

I am pretty fresh in machine learning so probably I made rookie mistake somewhere but I tried a lot last few days and I can't find a way to further improve my network. Hope you guys can help!

So I was asked to make "perceptron Xor gate" in Tensorflow with data that I generated myself. Since single perceptron can't solve it I used MLP. My generated train and test data looks like this:

# TRAIN SET

number_of_elements = 20000

top_right, top_left, bottom_right, bottom_left = [np.random.rand(number_of_elements, 2) for _ in range(4)]
top_left[:, 0] *= -1
bottom_right[1] *= -1
bottom_left * -1

X = np.concatenate((top_right, top_left, bottom_right, bottom_left))

top_right_y, bottom_left_y = np.zeros((2, number_of_elements))
top_left_y, bottom_right_y = np.ones((2, number_of_elements))

y = np.concatenate((top_right_y, top_left_y, bottom_right_y, bottom_left_y))
y = np.reshape(y, [y.shape[0], 1])

# TEST SET

number_of_test_elements = 40000

first_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)
second_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)
third_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)
forth_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)

top_right = np.random.rand(first_el_random, 2)
top_left = np.random.rand(second_el_random, 2)
bottom_right = np.random.rand(third_el_random, 2)
bottom_left = np.random.rand(forth_el_random, 2)
top_left[:, 0] *= -1
bottom_right[1] *= -1
bottom_left * -1
X_test_v1 = np.concatenate((bottom_right, bottom_left, top_left, top_right))

top_right_y = np.zeros(first_el_random)
top_left_y = np.ones(second_el_random)
bottom_right_y = np.ones(third_el_random)
bottom_left_y = np.zeros(forth_el_random)

y_test_v1 = np.concatenate((bottom_right_y, bottom_left_y, top_left_y, top_right_y))
y_test_v1 = np.reshape(y_test_v1, [y_test_v1.shape[0], 1])

Then I made two implementations, one in Keras and one in Tensorflow. They look almost the same so I will just show Keras code:

# KERAS MODEL

model = Sequential()
model.add(Dense(64, input_dim=2, activation='tanh', kernel_initializer='normal'))
model.add(Dropout(0.2))
model.add(Dense(32, activation='tanh', kernel_initializer='normal', kernel_constraint=maxnorm(3)))
model.add(Dropout(0.2))
model.add(Dense(16, activation='tanh', kernel_initializer='normal', kernel_constraint=maxnorm(3)))
model.add(Dense(1, activation='sigmoid', kernel_initializer='normal'))

optimize = Adam()
learning_rate_reduction = ReduceLROnPlateau(monitor='val_binary_accuracy', patience=10, verbose=2, factor=0.5, min_lr=0.00001)

model.compile(loss='binary_crossentropy', optimizer=optimize, metrics=['binary_accuracy'])
model.fit(X, y, batch_size=200, shuffle=True, epochs=100, verbose=2, validation_split=0.33,
          callbacks=[learning_rate_reduction])

preds = model.predict_classes(X_test_v1)

real_accuracy = np.mean(np.equal(y_test_v1, preds))
print('Test classification accuracy: %.4f' % real_accuracy)

Model accuracy is around 75%, I tried to make it higher by: - increasing train data size - adding additional dense layers, removing layers, adding dropout, removing dropout, adding neurons, removing neurons etc - changing activation functions, only one that I didn't change is sigmoid on outer layer, when I mess with that my results don't make sense - messing with different optimizers, learning rates, callbacks. I didn't touch binary_crossentropy as loss function tho. - changing bath sizes, epochs, setting shuffle to False

The problem seems pretty simple in general but I just can't pinpoint what am I doing wrong. If someone can tell me what am I doing wrong I would be really grateful. Thanks!

Answer 1

After carefully reviewing my code I found a bug in my train and test set.

Bug was in this line:

bottom_right[1] *= -1

should be:

bottom_right[:, 1] *= -1

After that my tensorflow accuracy went up to 99% and my Keras accuracy to 89%.

Anyway if someone has better solution or see something I am doing wrong please let me know :)

Answer 2

I'm pretty new in the field as well. However I maneged to get a 1.0 binary accuracy.

model = Sequential()
num_neu = 10
model.add(Dense(num_neu, input_dim=2, activation='sigmoid'))
model.add(Dense(num_neu, activation='sigmoid'))
model.add(Dense(num_neu, activation='sigmoid'))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='mean_squared_error',
              optimizer='adam',
              metrics=['binary_accuracy'])
history = model.fit(X, y, nb_epoch=500)

For the data generation, I simply used the make_blobs() method from scikit_learn