It then says: "The update rule is derived by computing the gradient for each element of the k-th mean and solving for the value where the gradient is zero." Can somebody tell me what this question means? I already understand how k-means clustering works, but I need some clarification about what it means by "k-th mean" and "gradient".
The k-th mean just means the mean of the k-th cluster. We want to minimize the sum of squares of each of the clusters, which we do by minimizing this so-called distortion metric. The gradient is a multivariate generalization of derivatives and the gradient of this distortion metric is the zero-vector when this is optimized, and after this you update your clusters.