Straight fromLeaky ReLUs allow a small, non-zero gradient when the unit is not active:
$$f(x) = \begin{cases} x & \text{if $x>0$}\\ \mathbf{0.01}x & \text{otherwise} \end{cases} $$
Parametric ReLUs take this idea further by making the coefficient of leakage wikipedia($0.01$ above) into a parameter that is learned along with the other neural network parameters:
$$f(x) = \begin{cases}
x & \text{if $x>0$}\\
\alpha x & \text{otherwise}
\end{cases} $$
Leaky ReLUs allow a small, non-zero gradient when the unit is not active.
Parametric ReLUs take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural network parameters.
Where $\alpha$ is the learnable parameter that is learned through gradient descent similar to the other neural network parameters such as weights and biases. Source
