# Is it possible to train a single input->neuron->relu->neuron->relu for input > 0.5?

The neural network is simply:

y=max(max(x*w+b,0)*v+d,0)


w,b is weight and bias of first neuron. v,d is weight and bias of second neuron.

If data is for example:

x = tensor([[1.0], [0.9], [0.8], [0.75], [0.7], [0.6], [0.51], [0.49], [0.3], [0.25], [0.2], [0.1], [0.0]])
y = tensor([[1.0], [1.0], [1.0], [1.0 ], [1.0], [1.0], [1.0 ], [0.0 ], [0.0], [0.0 ], [0.0], [0.0], [0.0]])


Then, below values fit the data:

w=-12
b=6
v=-12
d=1


Is it possible to train the network to find above values (or other possible values) ?

I tried below code (which actually works sometimes but fails most of the times):

l1 = nn.Linear(1, 1)
l2 = nn.Linear(1, 1)
relu1 = nn.ReLU()
relu2 = nn.ReLU()

x = tensor([[1.0], [0.9], [0.8], [0.75], [0.7], [0.6], [0.51], [0.49], [0.3], [0.25], [0.2], [0.1], [0.0]])
y = tensor([[1.0], [1.0], [1.0], [1.0 ], [1.0], [1.0], [1.0 ], [0.0 ], [0.0], [0.0 ], [0.0], [0.0], [0.0]])

lr = 0.5

for i in range(0, 100):
out = relu2(l2(relu1(l1(x))))
lss = F.mse_loss(out, y)
lss.backward()

relu2(l2(relu1(l1(x))))

• I also tried lss=F.binary_cross_entropy(torch.sigmoid(out), y)
– Tbon
Apr 8, 2023 at 23:09
• The main problem seems to be dying ReLUs. Exponential linear units (ELU) do help, but then cannot exactly represent the function with only 2 neurons I guess...
– Tbon
Apr 10, 2023 at 8:46
• Yes, having only two neurons is a problem. Also, you'd probably want more data points, and you'd want to shuffle your data and initialize your random seed. What you're describing is a Heaviside function - [link]en.wikipedia.org/wiki/Heaviside_step_function. The derivative of this is going to be zero everywhere except at x = 0, which means your gradients will go to zero pretty quickly. You can see this by printing out your gradients from within the training loop. Apr 10, 2023 at 11:32

target = torch.abs(torch.ceil((torch.zeros_like(x) - 0.5) - x))

• Ah, got it. I'd recommend starting with simple linear regression in that case - basically fitting a line to a set of potentially noisy data points. This can be as simple as class LR(nn.Module): def __init__(self, input_size, output_size): super().__init__() self.linear = nn.Linear(input_size, output_size) def forward(self, x): pred = self.linear(x) return pred Apr 10, 2023 at 9:44