Backpropagation stops when, as it tunes the model parameters, it finds a minimum in the cost function. At the minimum, the cost function has zero gradient, by definition. Numerically, the gradient will never be exactly zero, so that’s why the algorithm stops when the gradient falls below a certain minimum.
That’s a vanishing gradient, but it’s what should happen when looking for a minimum, so that’s not the vanishing gradient problem. The problem is when the gradient is very small, even when you’re not near the minimum. You haven’t tuned your parameters at all yet, and the performance of the model is terrible, but the gradient of the cost function is practically zero. That means you don’t know how to update your parameters, so you can’t train.
This happens, for example, in a regular feed-forward network, with many layers that all use the sigmoid activation function. The cost function is calculated using the output of the last layer. The gradient of the cost function as a function of the parameters in the first layer contains the product of the gradients in all layer layers. The gradient of the sigmoid is below 1 always, so you’re multiplying many numbers that are below 1, leading to a very small number. That’s the vanishing gradient problem.
This also explains why using a ReLU rather than sigmoid helps, and why this problem gets worse, the more layers you have.