Does high regression coefficient for Principal components that don't explain much variance imply that my data is not a good predictor?

Question

There isn't much to add to the question. Essentially i had some data that I reduced to 4 principal components, the first two components of which explain 99% of the variance in my data. Upon building a linear regression model using these 4 components, the coefficients for component 3 and 4 is significantly higher (by an order of 20) in magnitude than the components 1 and 2.

Without having to look at accuracy of the model, is this information sufficient to conclude that my data is a very weak linear predictor of the dependent variable?

Answer 1

The variance of the data is just that, variance. Imagine I'm trying to predict y from x1 and x2, and the true underlying model is y = x1 + e, where e~N(0,.001) and x1~N(0,.01), and x2 is drawn independently from N(0,1). There is significantly more variance in x2, but it is useless in predicting y, where x1 would be an extremely good predictor.

EDIT: We can implement this in R.

e = rnorm(100,0,.001) x1 = rnorm(100,0,.01) x2 = rnorm(100,0,1) y = x1 + e x = data.frame(cbind(x1,x2)) pca1 = prcomp(x) pca1 summary(pca1) model <- lm(y ~ ., data=data.frame(pca1$x))

Here is what our PCA looks like

PC1 PC2 x1 0.0003702204 -0.9999999315 x2 0.9999999315 0.0003702204 Importance of components: PC1 PC2 Standard deviation 1.0466 0.01098 Proportion of Variance 0.9999 0.00011 Cumulative Proportion 0.9999 1.00000

So component PC1 represents 99.99% of our variance. Wow it must be a good predictor! Let's look at our model.

Call: lm(formula = y ~ ., data = data.frame(pca1$x))

Coefficients: Estimate Std. Error t value Pr(>|t|)
(Intercept) 7.589e-04 1.017e-04 7.463 3.64e-11 *** PC1 3.561e-05 1.035e-04 0.344 0.732
PC2 1.002e+00 1.039e-02 96.424 < 2e-16 ***

Oh. It turns out it's useless for prediction.

Answer 2

Welcome to the site!

In short - yes, it is fair to say that components 1 and 2 are weaker linear predictors of your target.

PCA converts a set of possibly correlated variables into a set of linearly uncorrelated variables - it's a dimension reduction technique to weed out correlated features - and involves only the predictor variables.

It is perfectly reasonable to see that the principal components are not strong predictors of the target response, as they were formed without a view (i.e. independently) of the target response.