The variance of the data is just that, variance. Imagine I'm trying to predict y from x1 and x2, and the true underlying model is y = x1 + e, where e~N(0,.001) and x1~N(0,.01), and x2 is drawn independently from N(0,1). There is significantly more variance in x2, but it is useless in predicting y, where x1 would be an extremely good predictor.
EDIT:
We can implement this in R.
e = rnorm(100,0,.001)
x1 = rnorm(100,0,.01)
x2 = rnorm(100,0,1)
y = x1 + e
x = data.frame(cbind(x1,x2))
pca1 = prcomp(x)
pca1
summary(pca1)
model <- lm(y ~ ., data=data.frame(pca1$x))
Here is what our PCA looks like
PC1 PC2
x1 0.0003702204 -0.9999999315
x2 0.9999999315 0.0003702204
Importance of components:
PC1 PC2
Standard deviation 1.0466 0.01098
Proportion of Variance 0.9999 0.00011
Cumulative Proportion 0.9999 1.00000
So component PC1 represents 99.99% of our variance. Wow it must be a good predictor! Let's look at our model.
Call:
lm(formula = y ~ ., data = data.frame(pca1$x))
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 7.589e-04 1.017e-04 7.463 3.64e-11 ***
PC1 3.561e-05 1.035e-04 0.344 0.732
PC2 1.002e+00 1.039e-02 96.424 < 2e-16 ***
Oh. It turns out it's useless for prediction.