LeakyRelu allows a small gradient when the unit is not active (negative): 

$$ f(x) = alpha * x \:\text{for} \: x < 0, $$
$$f(x) = x \:\text{for}\: x >= 0.$$