0
$\begingroup$

The neural network is simply:

y=max(max(x*w+b,0)*v+d,0)

w,b is weight and bias of first neuron. v,d is weight and bias of second neuron.

If data is for example:

x = tensor([[1.0], [0.9], [0.8], [0.75], [0.7], [0.6], [0.51], [0.49], [0.3], [0.25], [0.2], [0.1], [0.0]])
y = tensor([[1.0], [1.0], [1.0], [1.0 ], [1.0], [1.0], [1.0 ], [0.0 ], [0.0], [0.0 ], [0.0], [0.0], [0.0]])

Then, below values fit the data:

w=-12
b=6
v=-12
d=1

Is it possible to train the network to find above values (or other possible values) ?

I tried below code (which actually works sometimes but fails most of the times):

l1 = nn.Linear(1, 1)
l2 = nn.Linear(1, 1)
relu1 = nn.ReLU()
relu2 = nn.ReLU()

x = tensor([[1.0], [0.9], [0.8], [0.75], [0.7], [0.6], [0.51], [0.49], [0.3], [0.25], [0.2], [0.1], [0.0]])
y = tensor([[1.0], [1.0], [1.0], [1.0 ], [1.0], [1.0], [1.0 ], [0.0 ], [0.0], [0.0 ], [0.0], [0.0], [0.0]])

lr = 0.5

for i in range(0, 100):
    out = relu2(l2(relu1(l1(x))))
    lss = F.mse_loss(out, y)
    lss.backward()

    with torch.no_grad():
        l1.weight -= l1.weight.grad * lr
        l1.bias -= l1.bias.grad * lr
        l2.weight -= l2.weight.grad * lr
        l2.bias -= l2.bias.grad * lr
        l1.zero_grad()
        relu1.zero_grad()
        l2.zero_grad()
        relu2.zero_grad()

relu2(l2(relu1(l1(x))))
Improve this question
$\endgroup$
3
  • $\begingroup$ I also tried lss=F.binary_cross_entropy(torch.sigmoid(out), y) $\endgroup$
    – Tbon
    Apr 8 at 23:09
  • $\begingroup$ The main problem seems to be dying ReLUs. Exponential linear units (ELU) do help, but then cannot exactly represent the function with only 2 neurons I guess... $\endgroup$
    – Tbon
    Apr 10 at 8:46
  • $\begingroup$ Yes, having only two neurons is a problem. Also, you'd probably want more data points, and you'd want to shuffle your data and initialize your random seed. What you're describing is a Heaviside function - [link]en.wikipedia.org/wiki/Heaviside_step_function. The derivative of this is going to be zero everywhere except at x = 0, which means your gradients will go to zero pretty quickly. You can see this by printing out your gradients from within the training loop. $\endgroup$
    – brewmaster321
    Apr 10 at 11:32

1 Answer 1

Reset to default
0
$\begingroup$

If I understand correctly, you can do this with the line

target = torch.abs(torch.ceil((torch.zeros_like(x) - 0.5) - x))

with no need to train a neural network. If this isn't what you're looking for please clarify what the desired output looks like. thanks.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you. Actually I'm trying to understand how to train neural networks and just picked up a trivial example task for it. I tried bigger networks too, more data points and other loss functions too, but I cannot make it learn this simple function consistently so I think I'm missing something in my network or training loop... $\endgroup$
    – Tbon
    Apr 10 at 0:30
  • $\begingroup$ Ah, got it. I'd recommend starting with simple linear regression in that case - basically fitting a line to a set of potentially noisy data points. This can be as simple as class LR(nn.Module): def __init__(self, input_size, output_size): super().__init__() self.linear = nn.Linear(input_size, output_size) def forward(self, x): pred = self.linear(x) return pred $\endgroup$
    – brewmaster321
    Apr 10 at 9:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.