I've heard that a multilayer perceptron can approximate any function arbitrarily exact, given enough neurons. I wanted to try it, so I wrote the following code:

#!/usr/bin/env python

"""Example for learning a regression."""

import tensorflow as tf
import numpy

def plot(xs, ys_truth, ys_pred):
    Plot the true values and the predicted values.

    xs : list
        Numeric values
    ys_truth : list
        Numeric values, same length as `xs`
    ys_pred : list
        Numeric values, same length as `xs`
    import matplotlib.pyplot as plt
    truth_plot, = plt.plot(xs, ys_truth, '-o', color='#00ff00')
    pred_plot, = plt.plot(xs, ys_pred, '-o', color='#ff0000')
    plt.legend([truth_plot, pred_plot],
               ['Truth', 'Prediction'],
               loc='upper center')

# Parameters
learning_rate = 0.1
momentum = 0.6
training_epochs = 1000
display_step = 100

# Generate training data
train_X = []
train_Y = []

# First simple test: a linear function
f = lambda x: x+4

# Second, more complicated test: x^2
# f = lambda x: x**2

for x in range(-20, 20):
train_X = numpy.asarray(train_X)
train_Y = numpy.asarray(train_Y)
n_samples = train_X.shape[0]

# Graph input
X = tf.placeholder(tf.float32)
reshaped_X = tf.reshape(X, [-1, 1])
Y = tf.placeholder("float")

# Create Model
W1 = tf.Variable(tf.truncated_normal([1, 100], stddev=0.1), name="weight")
b1 = tf.Variable(tf.constant(0.1, shape=[1, 100]), name="bias")
mul = tf.matmul(reshaped_X, W1)
h1 = tf.nn.sigmoid(mul) + b1
W2 = tf.Variable(tf.truncated_normal([100, 100], stddev=0.1), name="weight")
b2 = tf.Variable(tf.constant(0.1, shape=[100]), name="bias")
h2 = tf.nn.sigmoid(tf.matmul(h1, W2)) + b2
W3 = tf.Variable(tf.truncated_normal([100, 1], stddev=0.1), name="weight")
b3 = tf.Variable(tf.constant(0.1, shape=[1]), name="bias")
# identity as activation to get arbitrary output
activation = tf.matmul(h2, W3) + b3

# Minimize the squared errors
l2_loss = tf.reduce_sum(tf.pow(activation-Y, 2))/(2*n_samples)
optimizer = tf.train.MomentumOptimizer(learning_rate, momentum).minimize(l2_loss)

# Initializing the variables
init = tf.initialize_all_variables()

# Launch the graph
with tf.Session() as sess:

    # Fit all training data
    for epoch in range(training_epochs):
        for (x, y) in zip(train_X, train_Y):
            sess.run(optimizer, feed_dict={X: x, Y: y})

        # Display logs per epoch step
        if epoch % display_step == 0:
            cost = sess.run(l2_loss, feed_dict={X: train_X, Y: train_Y})
            print("cost=%s\nW1=%s" % (cost, sess.run(W1)))

    print("Optimization Finished!")
    print("cost=%s W1=%s" %
          (sess.run(l2_loss, feed_dict={X: train_X, Y: train_Y}),
           sess.run(W1)))  # "b2=", sess.run(b2)

    # Get output and plot it
    ys_pred = []
    ys_truth = []

    test_X = []
    for x in range(-40, 40):

    for x in test_X:
        ret = sess.run(activation, feed_dict={X: x})
    plot(train_X.tolist(), ys_truth, ys_pred)

This kind of works for linear functions (at least for the training data, not so much for the testing data outside of the range):

enter image description here

However, it doesn't work at all for non-linear function $x^2$:

enter image description here

Why does this neural network not work for such simple function approximation? What do I have to change to make the same network topology work for both functions?

  • 1
    $\begingroup$ Neural networks don't extrapolate well, so once outside the target range, it will not work. The NN here never "learns the function", instead it learns how to best (lowest error) approximate the function in the range given by the training data. The $y=x^2$ function should work though, and I would expect there is a problem with your NN implementation. However, there is too much unfamiliar code here for me to find it. $\endgroup$ Commented Dec 23, 2015 at 9:10
  • $\begingroup$ @NeilSlater Do you have a source / argument for your claim that NNs (there are lots of them, not only MLPs!) don't extrapolate well? - I use Google TensorFlow. The model is defined under "Create Model". I use $x$ as input, a hidden layer with 100 neurons and sigmoid activation, then a hidden layer with another 100 neurons and sigmoid activation, then a linear activation and a single neuron. I use MSE and momentum. $\endgroup$ Commented Dec 23, 2015 at 10:04
  • 1
    $\begingroup$ It's a well-known property of NNs. In general, they interpolate between training data, they do not extrapolate. You could deliberately fit the form of the function by choosing your architecture and activation functions to match the desired end result, but that defeats the "general function approximation". Here's a discussion: groups.google.com/forum/#!topic/comp.ai.neural-nets/CH4C63CzhVo $\endgroup$ Commented Dec 23, 2015 at 10:23
  • $\begingroup$ For people finding this and looking for function approximation: I've wrote a short blog post how to do function approximation with gaussian processes and sklearn: martin-thoma.com/function-approximation $\endgroup$ Commented Jan 19, 2016 at 19:16

1 Answer 1


This does not answere your question directly, but might include some helpful pointers:

In the recent Deep Residual Learning for Image Recognition paper it is written:

If one hypothesizes that multiple nonlinear layers can asymptotically approximate complicated functions[...]

This hypothesis, however, is still an open question. See [28].


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.