Input: $x \in \mathbb{R}^+$

Expected output:

$f(x) = \begin{cases}x &\text{for } x \leq 5\\10 - x &\text{for } x > 5\end{cases}$

So basically you have a neural network with one input neuron and one output neuron. Let's say the output has the identity as an activation.

What is the simplest network with ReLU activations that fits this function?


1 Answer 1


In order to find the network, I tried a couple of different small networks:

One hidden layer with two neurons

This would be the function

$$out = \max(a_1 x + b_1, 0) + \max(a_2x+ b_2, 0)$$

$$a_1 = 1, b_1 = 0, a_2=1, b_2 = -1$$

enter image description here

$$a_1 = 1, b_1 = -5, a_2=-1, b_2 = 5$$

enter image description here

2 hidden layers, 3 neurons in total

This is pretty close:

enter image description here

I guess the pattern is right, but the values are just not quite right. Also, I'm not totally sure if only one hidden layer with 3 neurons (or more) might also work.

I always include biases.

The solution

    / o (ReLU) \
IN o            o (linear)
    \ o (ReLU) /

It is basically the 1 hidden layer with 2 neurons, but also using the scaling (-1?) and the bias (+5?) of the output layer.

$$(-1) \cdot (\max(x - 5, 0) + \max(-x + 5, 0))+5$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.