In this example: https://www.tensorflow.org/tutorials/keras/basic_regression I was surprised to see a neural network's predicted values graph as a straight line. Isn't the purpose of neural networks to approximate non-linearities of data into it?

Please let me know what I am missing here.


Neural networks are very good function approximators. Hence, they can approximate a wide range of nonlinear functions. Remember that linear functions are easier to represent than nonlinear functions. Hence, the neural network will clearly be able to approximate a linear function. This can be easiest seen if we only use linear activation functions. But we can also use a nonlinear ReLU and still be able to arbitrarily approximate the function on a compact set.

The question that we should have is rather: Is it overkill to approximate a linear function with a neural network when a linear regression would do the job? The answer should be clear that you should rather use a linear regression instead of a neural network.

The given example just wants to demonstrate that even without knowing the relationship between our predictors and criterion (this is sometimes called domain knowledge) the neural network will still be able to approximate the function without the need for additional domain knowledge.

| improve this answer | |
  • $\begingroup$ Thanks for answering. Actually, I was expecting the line in the second-last figure to be at least a bit wiggly/curvy as opposed to perfectly straight line. I though there is something specially configured into the NN given in the link so that it works like a linear regressor. $\endgroup$ – KCK May 19 '19 at 9:21
  • $\begingroup$ btw, I agree that it definitely is an overkill to use NN as linear regressor. $\endgroup$ – KCK May 19 '19 at 9:23
  • $\begingroup$ is the linearity because the possible outputs are >0 and RELU is used? (As far as I understand, RELU is linear if the domain in either [−∞,0] or [0,∞]) $\endgroup$ – KCK May 19 '19 at 9:37
  • $\begingroup$ @KCK: I think you are mixing different concepts. The one concept is function approximation. I haven't looked at the link that you provided. But they are plotting predictions vs true values. The diagonal line is not the prediction but rather the line of a perfect fit. You can see that the true values and the predictions do not lie on the diagonal line. If you want to see the function predictors -> outputs you would need to take your model and calculate it for different inputs (works only for two inputs) and plot the surface of the function to see its shape. $\endgroup$ – MachineLearner May 19 '19 at 10:12
  • 2
    $\begingroup$ Good answer (+1) though "a linear function is a trivial case of a nonlinear function" doesn't quite make sense. A subset is not a trivial case of its complement. $\endgroup$ – John Coleman May 19 '19 at 12:13

The plot you are referring to is not the function produced by the neural network! It is the plot of real target variables vs predicted target variables. The straight line $y=x$ represents a perfect model.

It certainly is true that a neural network might produce a linear function in the end, but that doesn't happen in this case (as you should expect from the bivariate plots earlier on in the notebook).

For example, add the following to a new cell to view the model's output vs Weight (taking a 2D slice by setting the other variables to medians/modes):

slice_data = normed_test_data
for col in ['Cylinders', 'USA', 'Europe', 'Japan', 'Model Year']:
  slice_data[col] = slice_data[col].mode().iloc[0]
for col in ['Displacement', 'Horsepower', 'Acceleration']:
  slice_data[col] = slice_data[col].median()
slice_pred = model.predict(slice_data).flatten()
plt.scatter(slice_data['Weight'], slice_pred)

A copy of the notebook with that added

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.