# Does high regression coefficient for Principal components that don't explain much variance imply that my data is not a good predictor?

There isn't much to add to the question. Essentially i had some data that I reduced to 4 principal components, the first two components of which explain 99% of the variance in my data. Upon building a linear regression model using these 4 components, the coefficients for component 3 and 4 is significantly higher (by an order of 20) in magnitude than the components 1 and 2.

Without having to look at accuracy of the model, is this information sufficient to conclude that my data is a very weak linear predictor of the dependent variable?

The variance of the data is just that, variance. Imagine I'm trying to predict y from x1 and x2, and the true underlying model is y = x1 + e, where e~N(0,.001) and x1~N(0,.01), and x2 is drawn independently from N(0,1). There is significantly more variance in x2, but it is useless in predicting y, where x1 would be an extremely good predictor.

EDIT: We can implement this in R.

e = rnorm(100,0,.001) x1 = rnorm(100,0,.01) x2 = rnorm(100,0,1) y = x1 + e x = data.frame(cbind(x1,x2)) pca1 = prcomp(x) pca1 summary(pca1) model <- lm(y ~ ., data=data.frame(pca1$x))  Here is what our PCA looks like PC1 PC2 x1 0.0003702204 -0.9999999315 x2 0.9999999315 0.0003702204 Importance of components: PC1 PC2 Standard deviation 1.0466 0.01098 Proportion of Variance 0.9999 0.00011 Cumulative Proportion 0.9999 1.00000 So component PC1 represents 99.99% of our variance. Wow it must be a good predictor! Let's look at our model. Call: lm(formula = y ~ ., data = data.frame(pca1$x))

Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) 7.589e-04 1.017e-04 7.463 3.64e-11 *** PC1 3.561e-05 1.035e-04 0.344 0.732 PC2 1.002e+00 1.039e-02 96.424 < 2e-16 ***

Oh. It turns out it's useless for prediction.

• I'm sorry, but could you elaborate a little as far as a Principal Components' representation of data is concerned? While i understand variance is a measure of the spread of the data, isn't a good measure of the validity of a principal component the amount of variance that it explains? – BolshackLancer Jun 26 '18 at 5:06
• Edited with some working code to try to better explain. – Andy M Jun 26 '18 at 15:23

Welcome to the site!

In short - yes, it is fair to say that components 1 and 2 are weaker linear predictors of your target.

PCA converts a set of possibly correlated variables into a set of linearly uncorrelated variables - it's a dimension reduction technique to weed out correlated features - and involves only the predictor variables.

It is perfectly reasonable to see that the principal components are not strong predictors of the target response, as they were formed without a view (i.e. independently) of the target response.

• Ah but can i extend this to say that my predictor variables are likely very weak predictors of the target variable? – BolshackLancer Jun 26 '18 at 9:05
• I don't believe you can infer that, since each predictor is essentially a function of the principal components - e.g. predictor_m = f(prin_comp_1, prin_comp_2,..., prin_comp_n) - so one weak principal component does not mean the predictor is weak. – bradS Jun 27 '18 at 8:12