# What is the simplest ReLU network that can learn an upside-down v pattern?

Input: $x \in \mathbb{R}^+$

Expected output:

$f(x) = \begin{cases}x &\text{for } x \leq 5\\10 - x &\text{for } x > 5\end{cases}$

So basically you have a neural network with one input neuron and one output neuron. Let's say the output has the identity as an activation.

What is the simplest network with ReLU activations that fits this function?

In order to find the network, I tried a couple of different small networks:

### One hidden layer with two neurons

This would be the function

$$out = \max(a_1 x + b_1, 0) + \max(a_2x+ b_2, 0)$$

$$a_1 = 1, b_1 = 0, a_2=1, b_2 = -1$$

$$a_1 = 1, b_1 = -5, a_2=-1, b_2 = 5$$

## 2 hidden layers, 3 neurons in total

This is pretty close:

I guess the pattern is right, but the values are just not quite right. Also, I'm not totally sure if only one hidden layer with 3 neurons (or more) might also work.

I always include biases.

## The solution

    / o (ReLU) \
IN o            o (linear)
\ o (ReLU) /


It is basically the 1 hidden layer with 2 neurons, but also using the scaling (-1?) and the bias (+5?) of the output layer.

$$(-1) \cdot (\max(x - 5, 0) + \max(-x + 5, 0))+5$$