Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
In this example on tensorflow website, I was surprised to see a neural network's predicted values graph as a straight line. Isn't the purpose of neural networks to approximate non-linearities of data into it?
Please let me know what I am missing here.
Neural networks are very good function approximators. Hence, they can approximate a wide range of nonlinear functions. Remember that linear functions are easier to represent than nonlinear functions. Hence, the neural network will clearly be able to approximate a linear function. This can be easiest seen if we only use linear activation functions. But we can also use a nonlinear ReLU and still be able to arbitrarily approximate the function on a compact set.
The question that we should have is rather: Is it overkill to approximate a linear function with a neural network when a linear regression would do the job? The answer should be clear that you should rather use a linear regression instead of a neural network.
The given example just wants to demonstrate that even without knowing the relationship between our predictors and criterion (this is sometimes called domain knowledge) the neural network will still be able to approximate the function without the need for additional domain knowledge.
The plot you are referring to is not the function produced by the neural network! It is the plot of real target variables vs predicted target variables. The straight line $y=x$ represents a perfect model.
It certainly is true that a neural network might produce a linear function in the end, but that doesn't happen in this case (as you should expect from the bivariate plots earlier on in the notebook).
For example, add the following to a new cell to view the model's output vs Weight (taking a 2D slice by setting the other variables to medians/modes):
slice_data = normed_test_data
for col in ['Cylinders', 'USA', 'Europe', 'Japan', 'Model Year']:
slice_data[col] = slice_data[col].mode().iloc[0]
for col in ['Displacement', 'Horsepower', 'Acceleration']:
slice_data[col] = slice_data[col].median()
slice_pred = model.predict(slice_data).flatten()
plt.scatter(slice_data['Weight'], slice_pred)
Neural Networks are very flexible and with a proper architecture + training they can model just about anything. The reasons you don' see ANNs used everywhere is because
ANNs are indeed used more often in non-linear data because regression is much more efficient to use on linear data.
What this shows is that ANNs can solve just about any problem even if the programmer doesn't understand it which is where the saying "Neural Networks are always the second best answer to every problem" comes from; in this case understanding that this is a linear relation and using regression would be the best solution.
However, the picture you are referring to appears to be the showing the prediction error rather than a linearity; if the model was 100% accurate then all the points would be on the line, if the predicted value is lower than the real value the point appears above the line and if its higher it appears above it.
