When updating the weights of a deep neural network using backpropagation, to update the weights of a given hidden layer, we use both the partial derivatives of the objective function with respect to the weights of units on deeper layers (according to the chain rule) as well as the results of activations of units on more shallow layers. The deeper the layer, the more activations influence the update and the less gradients do.
On Wikipedia, the vanishing gradients or exploding gradients (which I personally call amplifying gradients) problem is said to affect "early layers". This makes sense, respectively for small or large gradients on later layers.
However, using rectified linear units (ReLU) is often cited as a solution to this, since the gradient is a crisp constant 1 for positive activations. Still, as can be seen on this visualization, a network which uses ReLU is still prone to amplifying gradients.
This time around, however, I would say vanishing gradients are caused by vanishing activations, in turn caused by initializing to values which are too small, and it is late layers, not early layers, which are affected. This answer seems to describe the same idea, referring to them as exploding values.
Am I wrong? Furthermore, in the linked visualization, too small initialization still leads to convergence, but too large initialization doesn't seem to at all... Why is that?