3
$\begingroup$

I'm trying to build a toy model which can identify a constant difference between two variables: (if variable1- variable2>10 then 1 else 0).

This should be a quite simple task for any regression model, but I want to solve it with NN. However all simple NN I built can not give me more than 51% accuracy.

Is this something that I do not understand?

My code:

seed = 7
np.random.seed(seed)

x1 = np.arange(50000)
x2 = x1+10+(0.5-np.random.rand(len(x1)))
X = np.column_stack((x1,x2))

Y = (x2-x1)>10

encoder = LabelEncoder()
encoder.fit(Y)
encoded_Y = encoder.transform(Y)

train_X, test_X, train_y, test_y = train_test_split(
    X, Y, train_size=0.9, random_state=0)

model = Sequential()
model.add(Dense(2, input_dim=2,activation='relu'))
model.add(Dense(16, activation='relu'))
model.add(Dense(1,  activation='sigmoid'))

model.compile(optimizer=RMSprop(lr=0.001), 
  loss=binary_crossentropy, metrics=[binary_accuracy])

history = model.fit(train_X, train_y, epochs=1000, batch_size=10,
  validation_data=(test_X, test_y), verbose=1)
$\endgroup$
5
$\begingroup$

Your problem is that neural networks work poorly when the input is not scaled to a simple range. A usual choice is to scale and offset each column so that it has mean 0 and standard deviation 1.

In your case, x1 and x2 vary from 0 to 49999 and roughly 10 to 50009. This range for inputs will causes lots of numeric issues.

With a balanced dataset as you have, 51% accuracy is basically just guessing (within experimental error), so the network has learned nothing. Try again with scaling - e.g.

x1 = (x1 - 25000) / 14433
x2 = (x2 - 25000) / 14433

I have tested your code with this addition, and it gains 100% validation accuracy within the first epoch.

If you want to assess other values in testing later you will need to scale them in the same way.

Note your predictions may be off in testing when x1 and x2 are not close to 10 apart, because you have only trained with examples which are close to exactly 10 apart. How the network behaves when this is not the case - e.g. for inputs of x1 = 100 and x2 = 1000, or x1 = 90 and x2 = 15, may not generalise well compared to the original comparison function.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.