0
$\begingroup$

I'm trying to create a model that, given a feature $x_i$, predicts $y_i$ such that $y_i=ax^2_i+bx_i+c$ by using backpropagation.

To do this, I'm using the ReLU activation function for each layer.

The output fits well when $x_i > 0$, but when it isn't it just outputs a flat line, as you can see in the picture:

enter image description here

Thinking about the reason, it seems pretty logical that when the output < 0 its derivative is also 0, hence the output is just the bias, but I cannot understand how it's possible to predict also negative values.

I'm not posting any code because I think the problem is more mathematical than due to programming, however if it's necessary I can post it.

$\endgroup$

3 Answers 3

1
$\begingroup$

I know that it seems to be obvious but did you omit to send negative x values to your model ? The behaviour of your model is pretty weird if you have linear layer as last layer. I tried to predict a polynomial function from backpropagation on my own and that's working. If you want, I can show you the code. I used a simple model:

model = nn.Sequential(
nn.Linear(1, 10),
nn.ReLU(),
nn.Linear(10, 12),
nn.ReLU(),
nn.Linear(12, 6),
nn.ReLU(),
nn.Linear(6, 1)

)

$\endgroup$
1
  • $\begingroup$ from what i remember the problem was that. lmao $\endgroup$
    – Iya Lee
    yesterday
0
$\begingroup$

In a regression problem, the last layer should not have an activation, it must be just a linear layer. Please check that this is the case

$\endgroup$
1
  • $\begingroup$ It doesn't, the graph is still the same. In case you're wondering, weights are initialized with np.random.randn() $\endgroup$
    – Iya Lee
    Mar 27 at 15:28
0
$\begingroup$

There are many possible ways to improve your model fit.

One possible way is to add a constant to the target so all target values are positive. Then invert this transformation to predict originally scaled values.

Another possible method is to add more features that capture the nonlinearities in the data.

$\endgroup$
1
  • $\begingroup$ I tried to add more layers and more features, but the result doesn't change. I don't think it's a problem of missing features, because the nonlinearities of the positive labels are captured. $\endgroup$
    – Iya Lee
    Mar 27 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.