There isn't much to add to the question. Essentially i had some data that I reduced to 4 principal components, the first two components of which explain 99% of the variance in my data. Upon building a linear regression model using these 4 components, the coefficients for component 3 and 4 is significantly higher (by an order of 20) in magnitude than the components 1 and 2.

Without having to look at accuracy of the model, is this information sufficient to conclude that my data is a very weak linear predictor of the dependent variable?


The variance of the data is just that, variance. Imagine I'm trying to predict y from x1 and x2, and the true underlying model is y = x1 + e, where e~N(0,.001) and x1~N(0,.01), and x2 is drawn independently from N(0,1). There is significantly more variance in x2, but it is useless in predicting y, where x1 would be an extremely good predictor.

EDIT: We can implement this in R.

e = rnorm(100,0,.001) x1 = rnorm(100,0,.01) x2 = rnorm(100,0,1) y = x1 + e x = data.frame(cbind(x1,x2)) pca1 = prcomp(x) pca1 summary(pca1) model <- lm(y ~ ., data=data.frame(pca1$x))

Here is what our PCA looks like

PC1 PC2 x1 0.0003702204 -0.9999999315 x2 0.9999999315 0.0003702204 Importance of components: PC1 PC2 Standard deviation 1.0466 0.01098 Proportion of Variance 0.9999 0.00011 Cumulative Proportion 0.9999 1.00000

So component PC1 represents 99.99% of our variance. Wow it must be a good predictor! Let's look at our model.

Call: lm(formula = y ~ ., data = data.frame(pca1$x))

Coefficients: Estimate Std. Error t value Pr(>|t|)
(Intercept) 7.589e-04 1.017e-04 7.463 3.64e-11 *** PC1 3.561e-05 1.035e-04 0.344 0.732
PC2 1.002e+00 1.039e-02 96.424 < 2e-16 ***

Oh. It turns out it's useless for prediction.

| improve this answer | |
  • $\begingroup$ I'm sorry, but could you elaborate a little as far as a Principal Components' representation of data is concerned? While i understand variance is a measure of the spread of the data, isn't a good measure of the validity of a principal component the amount of variance that it explains? $\endgroup$ – BolshackLancer Jun 26 '18 at 5:06
  • $\begingroup$ Edited with some working code to try to better explain. $\endgroup$ – Andy M Jun 26 '18 at 15:23

Welcome to the site!

In short - yes, it is fair to say that components 1 and 2 are weaker linear predictors of your target.

PCA converts a set of possibly correlated variables into a set of linearly uncorrelated variables - it's a dimension reduction technique to weed out correlated features - and involves only the predictor variables.

It is perfectly reasonable to see that the principal components are not strong predictors of the target response, as they were formed without a view (i.e. independently) of the target response.

| improve this answer | |
  • $\begingroup$ Ah but can i extend this to say that my predictor variables are likely very weak predictors of the target variable? $\endgroup$ – BolshackLancer Jun 26 '18 at 9:05
  • $\begingroup$ I don't believe you can infer that, since each predictor is essentially a function of the principal components - e.g. predictor_m = f(prin_comp_1, prin_comp_2,..., prin_comp_n) - so one weak principal component does not mean the predictor is weak. $\endgroup$ – bradS Jun 27 '18 at 8:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.