Regression model doesn't handle negative values

Question

I'm trying to create a model that, given a feature $x_i$, predicts $y_i$ such that $y_i=ax^2_i+bx_i+c$ by using backpropagation.

To do this, I'm using the ReLU activation function for each layer.

The output fits well when $x_i > 0$, but when it isn't it just outputs a flat line, as you can see in the picture:

Thinking about the reason, it seems pretty logical that when the output < 0 its derivative is also 0, hence the output is just the bias, but I cannot understand how it's possible to predict also negative values.

I'm not posting any code because I think the problem is more mathematical than due to programming, however if it's necessary I can post it.

Answer 1

I know that it seems to be obvious but did you omit to send negative x values to your model ? The behaviour of your model is pretty weird if you have linear layer as last layer. I tried to predict a polynomial function from backpropagation on my own and that's working. If you want, I can show you the code. I used a simple model:

model = nn.Sequential(
nn.Linear(1, 10),
nn.ReLU(),
nn.Linear(10, 12),
nn.ReLU(),
nn.Linear(12, 6),
nn.ReLU(),
nn.Linear(6, 1)

)

Answer 2

In a regression problem, the last layer should not have an activation, it must be just a linear layer. Please check that this is the case

Answer 3

There are many possible ways to improve your model fit.

One possible way is to add a constant to the target so all target values are positive. Then invert this transformation to predict originally scaled values.

Another possible method is to add more features that capture the nonlinearities in the data.