0
$\begingroup$

I'm trying to create a model that, given a feature $x_i$, predicts $y_i$ such that $y_i=ax^2_i+bx_i+c$ by using backpropagation.

To do this, I'm using the ReLU activation function for each layer.

The output fits well when $x_i > 0$, but when it isn't it just outputs a flat line, as you can see in the picture:

enter image description here

Thinking about the reason, it seems pretty logical that when the output < 0 its derivative is also 0, hence the output is just the bias, but I cannot understand how it's possible to predict also negative values.

I'm not posting any code because I think the problem is more mathematical than due to programming, however if it's necessary I can post it.

$\endgroup$

3 Answers 3

1
$\begingroup$

I know that it seems to be obvious but did you omit to send negative x values to your model ? The behaviour of your model is pretty weird if you have linear layer as last layer. I tried to predict a polynomial function from backpropagation on my own and that's working. If you want, I can show you the code. I used a simple model:

model = nn.Sequential(
nn.Linear(1, 10),
nn.ReLU(),
nn.Linear(10, 12),
nn.ReLU(),
nn.Linear(12, 6),
nn.ReLU(),
nn.Linear(6, 1)

)

$\endgroup$
3
  • $\begingroup$ from what i remember the problem was that. lmao $\endgroup$
    – Iya Lee
    Sep 26, 2023 at 13:59
  • $\begingroup$ Ok great ! I you want I can send you my whole code ! $\endgroup$ Oct 12, 2023 at 7:15
  • $\begingroup$ already solved months ago. thanks however $\endgroup$
    – Iya Lee
    Oct 15, 2023 at 17:05
0
$\begingroup$

In a regression problem, the last layer should not have an activation, it must be just a linear layer. Please check that this is the case

$\endgroup$
1
  • $\begingroup$ It doesn't, the graph is still the same. In case you're wondering, weights are initialized with np.random.randn() $\endgroup$
    – Iya Lee
    Mar 27, 2023 at 15:28
0
$\begingroup$

There are many possible ways to improve your model fit.

One possible way is to add a constant to the target so all target values are positive. Then invert this transformation to predict originally scaled values.

Another possible method is to add more features that capture the nonlinearities in the data.

$\endgroup$
1
  • $\begingroup$ I tried to add more layers and more features, but the result doesn't change. I don't think it's a problem of missing features, because the nonlinearities of the positive labels are captured. $\endgroup$
    – Iya Lee
    Mar 27, 2023 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.