0
$\begingroup$

The neural network is simply:

y=max(max(x*w+b,0)*v+d,0)

w,b is weight and bias of first neuron. v,d is weight and bias of second neuron.

If data is for example:

x = tensor([[1.0], [0.9], [0.8], [0.75], [0.7], [0.6], [0.51], [0.49], [0.3], [0.25], [0.2], [0.1], [0.0]])
y = tensor([[1.0], [1.0], [1.0], [1.0 ], [1.0], [1.0], [1.0 ], [0.0 ], [0.0], [0.0 ], [0.0], [0.0], [0.0]])

Then, below values fit the data:

w=-12
b=6
v=-12
d=1

Is it possible to train the network to find above values (or other possible values) ?

I tried below code (which actually works sometimes but fails most of the times):

l1 = nn.Linear(1, 1)
l2 = nn.Linear(1, 1)
relu1 = nn.ReLU()
relu2 = nn.ReLU()

x = tensor([[1.0], [0.9], [0.8], [0.75], [0.7], [0.6], [0.51], [0.49], [0.3], [0.25], [0.2], [0.1], [0.0]])
y = tensor([[1.0], [1.0], [1.0], [1.0 ], [1.0], [1.0], [1.0 ], [0.0 ], [0.0], [0.0 ], [0.0], [0.0], [0.0]])

lr = 0.5

for i in range(0, 100):
    out = relu2(l2(relu1(l1(x))))
    lss = F.mse_loss(out, y)
    lss.backward()

    with torch.no_grad():
        l1.weight -= l1.weight.grad * lr
        l1.bias -= l1.bias.grad * lr
        l2.weight -= l2.weight.grad * lr
        l2.bias -= l2.bias.grad * lr
        l1.zero_grad()
        relu1.zero_grad()
        l2.zero_grad()
        relu2.zero_grad()

relu2(l2(relu1(l1(x))))
$\endgroup$
3
  • $\begingroup$ I also tried lss=F.binary_cross_entropy(torch.sigmoid(out), y) $\endgroup$
    – Tbon
    Apr 8, 2023 at 23:09
  • $\begingroup$ The main problem seems to be dying ReLUs. Exponential linear units (ELU) do help, but then cannot exactly represent the function with only 2 neurons I guess... $\endgroup$
    – Tbon
    Apr 10, 2023 at 8:46
  • $\begingroup$ Yes, having only two neurons is a problem. Also, you'd probably want more data points, and you'd want to shuffle your data and initialize your random seed. What you're describing is a Heaviside function - [link]en.wikipedia.org/wiki/Heaviside_step_function. The derivative of this is going to be zero everywhere except at x = 0, which means your gradients will go to zero pretty quickly. You can see this by printing out your gradients from within the training loop. $\endgroup$ Apr 10, 2023 at 11:32

1 Answer 1

0
$\begingroup$

If I understand correctly, you can do this with the line

target = torch.abs(torch.ceil((torch.zeros_like(x) - 0.5) - x))

with no need to train a neural network. If this isn't what you're looking for please clarify what the desired output looks like. thanks.

$\endgroup$
2
  • $\begingroup$ Thank you. Actually I'm trying to understand how to train neural networks and just picked up a trivial example task for it. I tried bigger networks too, more data points and other loss functions too, but I cannot make it learn this simple function consistently so I think I'm missing something in my network or training loop... $\endgroup$
    – Tbon
    Apr 10, 2023 at 0:30
  • $\begingroup$ Ah, got it. I'd recommend starting with simple linear regression in that case - basically fitting a line to a set of potentially noisy data points. This can be as simple as class LR(nn.Module): def __init__(self, input_size, output_size): super().__init__() self.linear = nn.Linear(input_size, output_size) def forward(self, x): pred = self.linear(x) return pred $\endgroup$ Apr 10, 2023 at 9:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.