I think your confusion arises from mixing two different kinds of terms together here. Bias and variance are general concepts which can be measured and quantified in a number of different ways. A residual is a specific measurement of the differences between a predicted value and a true value.
Bias, loosely speaking, is how far away the average prediction is from the actual average. One way to measure it is in the difference of the means. You could also use difference of medians, difference in range, or several other calculations. To a get a complete picture of the bias of a model, you will want to look at several different measures.
Variance, when used informally in data science, is a property of single sets (whether predictions or true values). The variance of a model, loosely speaking, is how far from the average prediction a randomly selected prediction will be. It's very often assessed using cross-validation. You construct multiple models using slightly different training sets but the same algorithm and tuning parameters. You then calculate an evaluation metric for each model, and calculate the standard deviation of this evaluation over all your models. This gives you a sense of the "stability" of a given algorithm/parameter set when exposed to different training and testing sets.
(N.B. This can be confusing because there is a specific definition of "variance" used in statistics, $v = \sigma^2$. In data science, it's usually used more informally.)
Residuals are a specific quantity associated with a single prediction/true value set pair. You've got the right definition there. This makes it not a general concept, but instead a measurement that you can use to assess either bias or variance. They're also frequently used in fitting regression models and otherwise performing gradient descent-style optimization. The mean or median of a residual set can be a way to assess bias, while the standard deviation of a residual set can be used to assess a variance.
