# Why can't my neural network learn how to predict the squares of natural numbers?

I want my neural network to learn to predict the square $$n+1$$ number having $$n$$ number. I am considering a regression problem. That's what I'm doing:

from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Embedding, Dropout
import numpy as np

x = np.array([[int(i)] for i in range(1001)])
y = np.array([x*x for x in range(1001)])

model = Sequential()
model.add(Dense(100, activation = 'relu', input_dim = 1))

model.fit(x,y,epochs= 2500)
pred = model.predict([1001])
print(pred)


However, as a result, I get [[ 1000166.8125]] instead 1002001.

Update:

x = np.array([[int(i)] for i in range(80001)])
y = np.array([x*x for x in range(80001)])
print(x)
print(y)
model = Sequential()
model.add(Dense(20, activation = 'relu', input_dim = 1))

model.fit(x,y,epochs= 3000)
pred = model.predict([80001])
print(pred)
model.save_weights("test.h5")
model_json = model.to_json()
json_file = open("test.json", "w")
json_file.write(model_json)
json_file.close()


result: [[ 4.81360333e+09]]

• Vaalizaadeh already provided a good answer to your question. If you want to get a feeling for the most common hyperparameters of DNN's, I'd suggest, you also check out the Tensorflow Playground. Also, check out this answer. May 6, 2019 at 7:30
• I want to push the second link in @georg_un's answer; it's pretty much exactly this question. In particular, a neural net seems unlikely to do a good job when the test data is outside the range of the training data (your 10001). The NN is trying very hard to fit the parabolic curve inside the training window, but will ignore everything outside that window: it has no incentive to do that "right". ( (It will probably ultimately produce a linear function, extending what it does at the ends of your training window.) May 7, 2019 at 1:34

Decrease the number of hidden layers; you can omit the dense layer with $$50$$ neurons. Furthermore, train your network more. You should also provide more data. It is not much at the moment.

Your current architecture is very deep for such a relatively easy task. Consequently, it needs more train time. You can just decrease the size of the current model by diminishing the number of hidden layers and neurons. For instance, use the following setting to see how you can train very fast and have a good accuracy.

model = Sequential()
model.add(Dense(20, activation = 'relu', input_dim = 1))

• did everything as you said. A data set of 10.000 squares and your architecture. I get: 10001^2 = [[ 99884920.]] instead 100020001.How big should the data set be? Is it possible to solve this problem using machine learning methods?
– SoH
May 6, 2019 at 7:29
• Yes, you can. It is a very easy task. Try to train your network more and decrease the learning rate. The default value is relatively high. May 6, 2019 at 7:34
• updated question. I could not get a good result.
– SoH
May 6, 2019 at 11:13
• @SoH It took so much time to find out what is really occurring. please first take a look at here. I will update the answer showing you what really is happening. The provided link can help you. May 6, 2019 at 13:26
• thanks! Apparently, to achieve a normal result is impossible.
– SoH
May 6, 2019 at 16:17

Because neural networks with a sufficiently large hidden layer can approximate arbitrary functions only on compact sets (this is one of the first things you can learn when you try to read some literature about neural networks). Train your neural network on a range from 0 to 100 and then ask the square of 78.

The much more interesting problem of learning the algorithm of multiplication is a part of an active research area called algorithm learning, for example check this paper.

By its very nature, Neural Networks are used to only approximate one given function. So it will work quite well for tasks where you do not need very precise information to perform, for example to recognise a picture you do not need to actually know every details of the picture. However, in your case, you really want to know the exact value of a specific function F (here n-> n^2). Thus in general you can not do that, except if your NN already contains the function F in itself (or with parameters). For example, usually NN uses very transcendental functions as tanh, for which I do not see how can produce a polynomial for you.