In this example on tensorflow website, I was surprised to see a neural network's predicted values graph as a straight line. Isn't the purpose of neural networks to approximate non-linearities of data into it?

Please let me know what I am missing here.


3 Answers 3


Neural networks are very good function approximators. Hence, they can approximate a wide range of nonlinear functions. Remember that linear functions are easier to represent than nonlinear functions. Hence, the neural network will clearly be able to approximate a linear function. This can be easiest seen if we only use linear activation functions. But we can also use a nonlinear ReLU and still be able to arbitrarily approximate the function on a compact set.

The question that we should have is rather: Is it overkill to approximate a linear function with a neural network when a linear regression would do the job? The answer should be clear that you should rather use a linear regression instead of a neural network.

The given example just wants to demonstrate that even without knowing the relationship between our predictors and criterion (this is sometimes called domain knowledge) the neural network will still be able to approximate the function without the need for additional domain knowledge.

  • $\begingroup$ Thanks for answering. Actually, I was expecting the line in the second-last figure to be at least a bit wiggly/curvy as opposed to perfectly straight line. I though there is something specially configured into the NN given in the link so that it works like a linear regressor. $\endgroup$
    – KCK
    Commented May 19, 2019 at 9:21
  • $\begingroup$ btw, I agree that it definitely is an overkill to use NN as linear regressor. $\endgroup$
    – KCK
    Commented May 19, 2019 at 9:23
  • $\begingroup$ is the linearity because the possible outputs are >0 and RELU is used? (As far as I understand, RELU is linear if the domain in either [−∞,0] or [0,∞]) $\endgroup$
    – KCK
    Commented May 19, 2019 at 9:37
  • $\begingroup$ @KCK: I think you are mixing different concepts. The one concept is function approximation. I haven't looked at the link that you provided. But they are plotting predictions vs true values. The diagonal line is not the prediction but rather the line of a perfect fit. You can see that the true values and the predictions do not lie on the diagonal line. If you want to see the function predictors -> outputs you would need to take your model and calculate it for different inputs (works only for two inputs) and plot the surface of the function to see its shape. $\endgroup$ Commented May 19, 2019 at 10:12
  • 2
    $\begingroup$ Good answer (+1) though "a linear function is a trivial case of a nonlinear function" doesn't quite make sense. A subset is not a trivial case of its complement. $\endgroup$ Commented May 19, 2019 at 12:13

The plot you are referring to is not the function produced by the neural network! It is the plot of real target variables vs predicted target variables. The straight line $y=x$ represents a perfect model.

It certainly is true that a neural network might produce a linear function in the end, but that doesn't happen in this case (as you should expect from the bivariate plots earlier on in the notebook).

For example, add the following to a new cell to view the model's output vs Weight (taking a 2D slice by setting the other variables to medians/modes):

slice_data = normed_test_data
for col in ['Cylinders', 'USA', 'Europe', 'Japan', 'Model Year']:
  slice_data[col] = slice_data[col].mode().iloc[0]
for col in ['Displacement', 'Horsepower', 'Acceleration']:
  slice_data[col] = slice_data[col].median()
slice_pred = model.predict(slice_data).flatten()
plt.scatter(slice_data['Weight'], slice_pred)

A copy of the notebook with that added


Neural Networks are very flexible and with a proper architecture + training they can model just about anything. The reasons you don' see ANNs used everywhere is because

  1. ANNs are "black boxes", we don't really understand how they work contrary to simpler methods
  2. They are very computationally intensive

ANNs are indeed used more often in non-linear data because regression is much more efficient to use on linear data.

What this shows is that ANNs can solve just about any problem even if the programmer doesn't understand it which is where the saying "Neural Networks are always the second best answer to every problem" comes from; in this case understanding that this is a linear relation and using regression would be the best solution.

However, the picture you are referring to appears to be the showing the prediction error rather than a linearity; if the model was 100% accurate then all the points would be on the line, if the predicted value is lower than the real value the point appears above the line and if its higher it appears above it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.