3
$\begingroup$

I have been trying to apply a simple neural network using keras to predict a sequence of numbers and the rule is if the input integer is odd it should be 4 and if its even it should be 2. Yet the neural network gets stuck at a 60% accuracy rate. Anyone know a solution to this?

from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout

from keras.layers.normalization import BatchNormalization

from sklearn.model_selection import cross_val_score
from keras.wrappers.scikit_learn import KerasClassifier

import numpy as np


def gen(x):
    if (x%2==0):
        return 2;
    else:
        return 4;


a = []           
for i in range(1,100001):
    a.append([i,gen(i)])

a = np.array(a)


x = a[:,0:1]
y = a[:,1:2]


def MakeClassifier():
    network_classifier = Sequential()
    network_classifier.add(Dense(units=2,kernel_initializer="uniform",activation="relu",input_dim=1)) #Hidden Layer1 taking into account number of inputs(independant variables(x)
    network_classifier.add(BatchNormalization())
    network_classifier.add(Dense(units=1,kernel_initializer="uniform",activation="sigmoid"))#OutPutLayer

    network_classifier.compile(optimizer="adam",loss="binary_crossentropy",metrics=["accuracy"])#If multicategorical then categorical_crossentropy
    return network_classifier

classifier = KerasClassifier(build_fn= MakeClassifier , batch_size = 10 , epochs = 1000)

classifier.fit(x,y,epochs=100,batch_size=1000)

print(classifier.predict([[6],[7]])) #Should Predict 2 and 4
$\endgroup$
2
  • 1
    $\begingroup$ 60% on training set or test? $\endgroup$ Commented Dec 23, 2017 at 8:26
  • $\begingroup$ It was on my train set and the test set accuracy score is roughly the same. $\endgroup$
    – Sam Thomas
    Commented Dec 23, 2017 at 8:42

1 Answer 1

2
$\begingroup$

There are two possible reasons for this result:

  • Low number of training examples
  • Using dense layers without batch normalization

You have a relatively deep network and your training set size is small. In such cases if you run your model too many times you will certainly overfit the training data. The reason is that whenever you have a powerful model that can learn complicated functions and if you provide low number of training examples, it has capability to fit the data and not learn it. Suppose that you have a fitting problem in calculus and you have 4 points and you also have polynomials with degree of four. In such cases you can fit the data exactly without any error but the point is that you are fitting it, not learning it. In your case your model is powerful and it tries to fit the data, not to learn it. The cure is to provide enough training data.

The reason that your model can not learn is that you are stacking dense layers without batch normalization. each time you update the weights, the output of the deep layers change and this cause the covariat shift. To avoid this problem use batch normalization.

As a solution for you I recommend you to putting just two hidden layers and manipulate the number of units in those layers and provide more training examples. Your task can easily be learned by a two hidden-layer-network.


After struggling for about one day finally I want to express my opinion about the problem. Depending on the problems, they may be solved using machine-learning or other techniques. Machine learning problems are those which you can not define an appropriate function to map the input to output or it may be so much hard to do so. Whenever you have the function, it can be used as the hypothesis, the final goal of machine learning algorithms. I tried hard and put so much time on this code and I get the same result. Nothing is going to be learned at all. I have been trying for about one day and still no progress has been seen. To explain the reason, the data is so much hard to be learned! for imagining how difficult it is, I recommend you to write numbers from one to ten in a straight line and put a line between consecutive numbers. The numbers are endless, so you will have no generalization because the boundaries that are going to be found will work just for the two neighbor numbers. This means that if you use the current features, you can not separate, learn, your data. I tried to do, somehow, feature engineering and used the following code to solve the problem:

import keras
import numpy as np
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers.normalization import BatchNormalization 
from keras import regularizers 
from keras.optimizers import Adam

def gen(x):
    if (x % 2 == 0):
        return 0; # represents 2
    else:
        return 1; # represents 4


a = []           
for i in range(1,100001):
    temp = np.random.randint(0, 10000000)
    a.append([temp, temp ** 2, temp ** 3, temp ** 4, gen(temp)])

a = np.array(a)


x = a[:, 0: a.shape[1] - 1]
y = a[:, a.shape[1] - 1:]

mean_of_x = np.mean(x, axis = 0, keepdims = True)
std_of_x = np.std(np.float64(x), axis = 0, keepdims = True)
x = (x - mean_of_x) / std_of_x

n_classes = 2
y = keras.utils.to_categorical(y, 2)

percentage = 95 / 100
limit = int(percentage * x.shape[0])

x_train = x[: limit, :]
y_train = y[: limit, :]

x_test = x[limit: , :]
y_test = y[limit: , :]

x_train.shape

model = Sequential()
model.add(Dense(1000, activation='relu', input_shape=(x_train.shape[1],)))
model.add(BatchNormalization())
# model.add(Dropout(0.5))
model.add(Dense(1000, activation='relu'))
model.add(BatchNormalization())
# model.add(Dropout(0.5))

model.add(Dense(n_classes, activation='softmax'))

model.summary()

model.compile(loss = 'categorical_crossentropy', optimizer = keras.optimizers.Adam(lr = 0.0001, decay = 1.5), metrics=['accuracy'])

model.fit(x_train, y_train, batch_size = 128, epochs = 2000, verbose = 1, validation_data=(x_test, y_test), shuffle = True,
          class_weight = {0: 10, 1: 1})

As you can see, the above code uses high order polynomial. Surprisingly, no progress was seen here too. this model has the mentioned problem for the previous feature in higher dimensions. That's why the learning does not happen here too.

the point here is that although you can not learn using the current feature, number itself, or its high order polynomials, you already have a solution for the problem. Instead of passing the number itself to the learning problem, pass the modulus two of the current number. This feature is so much easy to be learned. you may need just one unit.

$\endgroup$
6
  • $\begingroup$ It seems to have no luck even after changing the layers to just 2 and adding batch normalization layer. I also made a loop to generate 10000 examples based on my rule. Also tried changing number of neurons. $\endgroup$
    – Sam Thomas
    Commented Dec 24, 2017 at 6:20
  • $\begingroup$ Oh, my bad sorry. I have updated it now. $\endgroup$
    – Sam Thomas
    Commented Dec 26, 2017 at 5:24
  • $\begingroup$ No, I have changed the code posted above and tested it with 2 to 4 layers. Also messed with units parameter and none seems to work. $\endgroup$
    – Sam Thomas
    Commented Dec 26, 2017 at 11:40
  • 1
    $\begingroup$ I am traveling and I logged in just to approve your answer and say thank you for all that effort , your explanation makes sense and I really appreciate all your efforts. $\endgroup$
    – Sam Thomas
    Commented Dec 27, 2017 at 17:01
  • $\begingroup$ you ever heard about the XOR problem en.wikipedia.org/wiki/Perceptron#History? $\endgroup$
    – Valentas
    Commented Feb 9, 2018 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.