Why can't my neural network learn how to predict the squares of natural numbers?

Question

I want my neural network to learn to predict the square $n+1$ number having $n$ number. I am considering a regression problem. That's what I'm doing:

from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Dense, Embedding, Dropout
import numpy as np

x = np.array([[int(i)] for i in range(1001)])
y = np.array([x*x for x in range(1001)])

model = Sequential()
model.add(Dense(100, activation = 'relu', input_dim = 1))
model.add(Dense(50, activation = 'relu'))
model.add(Dense(10, activation = 'relu'))
model.add(Dense(1))

model.compile(loss='mse',optimizer='adam', metrics=['mae'])
model.fit(x,y,epochs= 2500)
pred = model.predict([1001])
print(pred)

However, as a result, I get [[ 1000166.8125]] instead 1002001.

Update:

x = np.array([[int(i)] for i in range(80001)])
y = np.array([x*x for x in range(80001)])
print(x)
print(y)
model = Sequential()
model.add(Dense(20, activation = 'relu', input_dim = 1))
model.add(Dense(20, activation = 'relu'))
model.add(Dense(1))

adam = optimizers.Adam(lr=0.0002,beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False,)
model.compile(loss='mse',optimizer=adam, metrics=['mae'])
model.fit(x,y,epochs= 3000)
pred = model.predict([80001])
print(pred)
model.save_weights("test.h5")
model_json = model.to_json()
json_file = open("test.json", "w")
json_file.write(model_json)
json_file.close()

result: [[ 4.81360333e+09]]

Answer 1

Decrease the number of hidden layers; you can omit the dense layer with $50$ neurons. Furthermore, train your network more. You should also provide more data. It is not much at the moment.

Your current architecture is very deep for such a relatively easy task. Consequently, it needs more train time. You can just decrease the size of the current model by diminishing the number of hidden layers and neurons. For instance, use the following setting to see how you can train very fast and have a good accuracy.

model = Sequential()
model.add(Dense(20, activation = 'relu', input_dim = 1))
model.add(Dense(20, activation = 'relu'))
model.add(Dense(1))

Answer 2

Because neural networks with a sufficiently large hidden layer can approximate arbitrary functions only on compact sets (this is one of the first things you can learn when you try to read some literature about neural networks). Train your neural network on a range from 0 to 100 and then ask the square of 78.

The much more interesting problem of learning the algorithm of multiplication is a part of an active research area called algorithm learning, for example check this paper.

Answer 3

By its very nature, Neural Networks are used to only approximate one given function. So it will work quite well for tasks where you do not need very precise information to perform, for example to recognise a picture you do not need to actually know every details of the picture. However, in your case, you really want to know the exact value of a specific function F (here n-> n^2). Thus in general you can not do that, except if your NN already contains the function F in itself (or with parameters). For example, usually NN uses very transcendental functions as tanh, for which I do not see how can produce a polynomial for you.