why multiplication (squares) doesn't work for neural networks?

Question

Below code creates the sum of 2 random numbers and then we train for 1000 examples and then we are able to predict which works fine.

Consider the below code for creating random data :

def random_sum_pairs(n_examples, n_numbers, largest):
    X, y = list(), list()
    for i in range(n_examples):
        in_pattern = [randint(1,largest) for _ in range(n_numbers)]
        out_pattern = sum(in_pattern)
        X.append(in_pattern)
        y.append(out_pattern)
    # format as NumPy arrays
    X,y = array(X), array(y)
    # normalize
    X = X.astype('float') / float(largest * n_numbers)
    y = y.astype('float') / float(largest * n_numbers)
    return X, y
 
# invert normalization
def invert(value, n_numbers, largest):
    return round(value * float(largest * n_numbers))

training the model :

n_examples = 1000
n_numbers = 2
largest = 1000

n_batch = 100
n_epoch = 500

model = Sequential()
model.add(Dense(20, input_dim=n_numbers))
model.add(Dense(100, input_dim=n_numbers))
model.add(Dense(1000, input_dim=n_numbers))
model.add(Dense(100, input_dim=n_numbers))
model.add(Dense(20))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer=adam)

X, y = random_sum_pairs(n_examples, n_numbers, largest)
model.fit(X, y, epochs=n_epoch, batch_size=n_batch, verbose=2)

predicting the model with:

result = model.predict(X, batch_size=n_batch, verbose=0)
# calculate error
expected = [invert(x, n_numbers, largest) for x in y]
predicted = [invert(x, n_numbers, largest) for x in result[:,0]]
rmse = sqrt(mean_squared_error(expected, predicted))
print('RMSE: %f' % rmse)
# show some examples
for i in range(20):
    error = expected[i] - predicted[i]
    print('Expected=%d, Predicted=%d (err=%d)' % (expected[i], predicted[i], error))

Result:

RMSE: 0.000000
Expected=120, Predicted=120 (err=0)
Expected=353, Predicted=353 (err=0)
Expected=1316, Predicted=1316 (err=0)
Expected=839, Predicted=839 (err=0)
Expected=731, Predicted=731 (err=0)
Expected=867, Predicted=867 (err=0)
Expected=276, Predicted=276 (err=0)
Expected=36, Predicted=36 (err=0)
Expected=601, Predicted=601 (err=0)
Expected=1805, Predicted=1805 (err=0)
Expected=1045, Predicted=1045 (err=0)
Expected=422, Predicted=422 (err=0)
Expected=1795, Predicted=1795 (err=0)
Expected=861, Predicted=861 (err=0)
Expected=469, Predicted=469 (err=0)
Expected=362, Predicted=362 (err=0)
Expected=119, Predicted=119 (err=0)
Expected=1021, Predicted=1021 (err=0)

But let's say I change the logic in random_sum_pairs to provide single numbers and squares of those numbers: (and change n_numbers = 1)

def random_sum_pairs(n_examples, n_numbers, largest):
  X, y = list(), list()
  for i in range(n_examples):
    in_pattern = [randint(1,largest) for _ in range(n_numbers)]
    #print(in_pattern)
    out_pattern = in_pattern[0]*in_pattern[0]
    #print(out_pattern)
    X.append(in_pattern)
    y.append(out_pattern)
    # format as NumPy arrays
  X,y = array(X), array(y)
  # normalize
  X = X.astype('float') / float(largest * largest)
  y = y.astype('float') / float(largest * largest)
  return X, y

This doesn't work at all and errors are huge. Results:

RMSE: 75777.312879
Expected=556516, Predicted=567106 (err=-10590)
Expected=403225, Predicted=458394 (err=-55169)
Expected=86436, Predicted=124424 (err=-37988)
Expected=553536, Predicted=565147 (err=-11611)
Expected=518400, Predicted=541642 (err=-23242)
Expected=927369, Predicted=779632 (err=147737)
Expected=855625, Predicted=742415 (err=113210)
Expected=159201, Predicted=227260 (err=-68059)
Expected=48841, Predicted=52929 (err=-4088)
Expected=71289, Predicted=97981 (err=-26692)
Expected=363609, Predicted=427054 (err=-63445)
Expected=116964, Predicted=171435 (err=-54471)
Expected=5476, Predicted=-91040 (err=96516)
Expected=316969, Predicted=387879 (err=-70910)
Expected=900601, Predicted=765921 (err=134680)
Expected=839056, Predicted=733601 (err=105455)

Why does this happen? I mean then for linear operations like summation, we don't even require neural networks and neural network is failing in a simple case like above squares of numbers, so how to train a neural network for learning the squares of numbers? I am not looking for 100% accurate results, but at least somewhat closer I was expecting.

Note: I know we don't need that dense network with those many hidden layers (I guess). I have tried with a single hidden layer as well, with similar results.

Answer 1

The reason you cannot fit non-linear functions (here sums of squares) is simply that your neural network is actually not a proper neural network: it simply resolves to a single linear element.

Why is that? Recall from the Keras documentation the Dense layer:

keras.layers.Dense(units, activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None)

where

activation: Activation function to use (see activations). If you don't specify anything, no activation is applied (ie. "linear" activation: a(x) = x).

Since you don't specify explicitly any activation, you actually use a linear one for all your layers. And it is well-known that a neural network comprised simply of linear units is equivalent to a simple linear unit (check Andrew Ng's lecture Why Non-linear Activation Functions for a detailed explanation); in fact, it is only with non-linear activation functions that neural networks begin to be able to do interesting things.

So, you should add activation='relu' in all your layers except the final one, which should remain as is (final layers in regression settings, like here, need linear activation functions).

You may also find the discussion in the Stack Overflow thread Is deep learning bad at fitting simple non-linear functions outside training scope (extrapolating)? interesting.

UPDATE (after comment):

• Determining the argument input_dim for any layer other than the first one is meaningless (the input dimension of layer N is simply the output dimension of layer N-1). Remove all input_dim arguments except in the first layer.

• Start simple: In the linked SO thread, there is a clear demonstration of a much simpler network that can indeed calculate the square of its input. There is a good chance that 500 epochs are not enough for the weights of your (unnecessarily large) network to converge (this was not the problem before because, as said, your network was essentially a simple linear unit).

• Additionally, you now seem to have an input you never use (in_pattern[1]); this can lead to further delay in the convergence of your model.

Please keep in mind that there is never a guarantee that any NN model can do a specific job, and experimenting with the architecture and the hyperparameters is always expected.