1
$\begingroup$

I am pretty fresh in machine learning so probably I made rookie mistake somewhere but I tried a lot last few days and I can't find a way to further improve my network. Hope you guys can help!

So I was asked to make "perceptron Xor gate" in Tensorflow with data that I generated myself. Since single perceptron can't solve it I used MLP. My generated train and test data looks like this:

# TRAIN SET

number_of_elements = 20000

top_right, top_left, bottom_right, bottom_left = [np.random.rand(number_of_elements, 2) for _ in range(4)]
top_left[:, 0] *= -1
bottom_right[1] *= -1
bottom_left * -1

X = np.concatenate((top_right, top_left, bottom_right, bottom_left))

top_right_y, bottom_left_y = np.zeros((2, number_of_elements))
top_left_y, bottom_right_y = np.ones((2, number_of_elements))

y = np.concatenate((top_right_y, top_left_y, bottom_right_y, bottom_left_y))
y = np.reshape(y, [y.shape[0], 1])

# TEST SET

number_of_test_elements = 40000

first_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)
second_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)
third_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)
forth_el_random = random.randint(number_of_test_elements / 2, number_of_elements * 2)

top_right = np.random.rand(first_el_random, 2)
top_left = np.random.rand(second_el_random, 2)
bottom_right = np.random.rand(third_el_random, 2)
bottom_left = np.random.rand(forth_el_random, 2)
top_left[:, 0] *= -1
bottom_right[1] *= -1
bottom_left * -1
X_test_v1 = np.concatenate((bottom_right, bottom_left, top_left, top_right))

top_right_y = np.zeros(first_el_random)
top_left_y = np.ones(second_el_random)
bottom_right_y = np.ones(third_el_random)
bottom_left_y = np.zeros(forth_el_random)

y_test_v1 = np.concatenate((bottom_right_y, bottom_left_y, top_left_y, top_right_y))
y_test_v1 = np.reshape(y_test_v1, [y_test_v1.shape[0], 1])

Then I made two implementations, one in Keras and one in Tensorflow. They look almost the same so I will just show Keras code:

# KERAS MODEL

model = Sequential()
model.add(Dense(64, input_dim=2, activation='tanh', kernel_initializer='normal'))
model.add(Dropout(0.2))
model.add(Dense(32, activation='tanh', kernel_initializer='normal', kernel_constraint=maxnorm(3)))
model.add(Dropout(0.2))
model.add(Dense(16, activation='tanh', kernel_initializer='normal', kernel_constraint=maxnorm(3)))
model.add(Dense(1, activation='sigmoid', kernel_initializer='normal'))

optimize = Adam()
learning_rate_reduction = ReduceLROnPlateau(monitor='val_binary_accuracy', patience=10, verbose=2, factor=0.5, min_lr=0.00001)

model.compile(loss='binary_crossentropy', optimizer=optimize, metrics=['binary_accuracy'])
model.fit(X, y, batch_size=200, shuffle=True, epochs=100, verbose=2, validation_split=0.33,
          callbacks=[learning_rate_reduction])

preds = model.predict_classes(X_test_v1)

real_accuracy = np.mean(np.equal(y_test_v1, preds))
print('Test classification accuracy: %.4f' % real_accuracy)

Model accuracy is around 75%, I tried to make it higher by: - increasing train data size - adding additional dense layers, removing layers, adding dropout, removing dropout, adding neurons, removing neurons etc - changing activation functions, only one that I didn't change is sigmoid on outer layer, when I mess with that my results don't make sense - messing with different optimizers, learning rates, callbacks. I didn't touch binary_crossentropy as loss function tho. - changing bath sizes, epochs, setting shuffle to False

The problem seems pretty simple in general but I just can't pinpoint what am I doing wrong. If someone can tell me what am I doing wrong I would be really grateful. Thanks!

$\endgroup$
0
$\begingroup$

After carefully reviewing my code I found a bug in my train and test set.

Bug was in this line:

bottom_right[1] *= -1

should be:

bottom_right[:, 1] *= -1

After that my tensorflow accuracy went up to 99% and my Keras accuracy to 89%.

Anyway if someone has better solution or see something I am doing wrong please let me know :)

$\endgroup$
0
$\begingroup$

I'm pretty new in the field as well. However I maneged to get a 1.0 binary accuracy.

model = Sequential()
num_neu = 10
model.add(Dense(num_neu, input_dim=2, activation='sigmoid'))
model.add(Dense(num_neu, activation='sigmoid'))
model.add(Dense(num_neu, activation='sigmoid'))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='mean_squared_error',
              optimizer='adam',
              metrics=['binary_accuracy'])
history = model.fit(X, y, nb_epoch=500)

For the data generation, I simply used the make_blobs() method from scikit_learn

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.