The variance of the data is just that, variance. Imagine I'm trying to predict y from x1 and x2, and the true underlying model is y = x1 + e, where e~N(0,.001) and x1~N(0,.01), and x2 is drawn independently from N(0,1). There is significantly more variance in x2, but it is useless in predicting y, where x1 would be an extremely good predictor.
We can implement this in R.
e = rnorm(100,0,.001)
x1 = rnorm(100,0,.01)
x2 = rnorm(100,0,1)
y = x1 + e
x = data.frame(cbind(x1,x2))
pca1 = prcomp(x)
model <- lm(y ~ ., data=data.frame(pca1$x))
Here is what our PCA looks like
x1 0.0003702204 -0.9999999315
x2 0.9999999315 0.0003702204
Importance of components:
Standard deviation 1.0466 0.01098
Proportion of Variance 0.9999 0.00011
Cumulative Proportion 0.9999 1.00000
So component PC1 represents 99.99% of our variance. Wow it must be a good predictor! Let's look at our model.
lm(formula = y ~ ., data = data.frame(pca1$x))
Estimate Std. Error t value Pr(>|t|)
(Intercept) 7.589e-04 1.017e-04 7.463 3.64e-11 ***
PC1 3.561e-05 1.035e-04 0.344 0.732
PC2 1.002e+00 1.039e-02 96.424 < 2e-16 ***
Oh. It turns out it's useless for prediction.