Quadratic regression with TensorFlow is not working

Question

I want to make a TensorFlow model that, given features $x$ and labels $y$ such that $y_i = ax_i^2+bx_i+c$, predicts reasonably well the equation.

x = np.arange(-1000, 1000, 0.74)
y = 1.3*x**2 + 5.3*x + 4

Now, here is the model:

model = tf.keras.Sequential([
    tf.keras.layers.Dense(64),
    tf.keras.layers.Dense(16),
    tf.keras.layers.Dense(1, activation="relu"),
])

model.compile(loss="mae", optimizer=tf.keras.optimizers.Adam(learning_rate = 0.01))
history = model.fit(tf.expand_dims(x, axis=-1), y, epochs = 100)

However, the model does not predict a parabola, but a straight line, as you see in the picture:

I've tried to add more layers, to increase or decrease the learning rate, but nothing sorts any kind of effect.

How can I fix it? Thanks.

Answer 1

You did not specify any activation function in your dense layers. When you stack multiple linear layers without any activation function, the end result is equivalent to a single linear layer (see this other answer for the mathematical proof). Therefore, the functions your network can learn are basically linear, with a final rectification from the ReLU.

You may add ReLU activations (or other activation, like tanh or sigmoid) to each of the intermediate layers to enable your network to model non-linear functions.

I would like to point out that when someone says "quadratic regression", it usually means that you are doing linear regression with extra variables computed by squaring the other variables. This is not what you are doing. You are modelling the quadratic function as if it was a black box, without using your knowledge of the underlying process.