I have a dataset and need to predict, out of 9 variables, which ones matter most to predict number 10.

I first tried using the selectKBestmethod from sklearn.feature_selection and it looks like 4,5,6 and 8 are best.

Then I tried to look at linear correlation using pearsonr between each of the variables from 1 to 9 and number 10, and I get 2,4,5,8 are most correlated.

However, when I used PCA from sklearn.decomposition and calculate the pca.explained_variance_ I get these values [ 2.13128046e+01 3.44315766e-01 3.26052258e-01 2.67148345e-01 1.85871921e-01 1.55241540e-01 1.31994073e-01 9.34982387e-02 1.03297667e-02]

Isn't it a problem that the first variable is so much higher than all the other ones? What does it mean?


2 Answers 2


Let's use the following notation : $(x_1, ...x_9)$ are the first 9 variables out of which you try to predict $y$, your tenth variable.

I will try to address what looks like a PCA misunderstanding, then give you some ways to predict which variables $ x_i $ matter most to predict $y$

Principal Component Analysis

The returned features you get from the PCA are not the original one!

The PCA transforms $(x_1, ..., x_9)$ into $(x'_1, ..., x'_9)$ where each $x'_i$ is a particular combination of the $(x_1, ..., x_9)$ such that for any $i, i'$, $x_i$ and $x_i'$ are linearly uncorrelated (ie orthogonal). Moreover, it is done such that each $x_i$ explains a certain part of the training dataset variance. In fact, most of the implementation sort the $x'_1, ..., x'_9$ such that the first one is the one explaining the variance the most, then the second, ...

Thus you end up having your first variable $x'_1$ explaining a lot your prediction $y$. But this $x'_i$ is a particular combinaison of $(x_1, ..., x_9)$.

Getting the feature importance

As to assert which variable $x_i$ explains $y$ the best, one might use different ways. Just to mention that the feature importance is not absolute ; it relies on the technique (or estimator) you use to address this question. Here is a non exhaustive list of possibilities :

  • The linear correlation is a good start
  • With the RandomForestClassifier (or RandomForestRegressor depending on $y$) of sklearn.ensemble, you can use feature_importances_ method to get which one is used the most at tree nodes. Note : it works with any decision tree estimator.
  • With standard linear or logistic regression (same with Lasso, Ridge, ...), you can check which variable has the higher coefficient (do not forget to normalize your input variables $(x_1, ..., x_9)$ )

I don't know how it is done using neural networks but would be glad if anyone has an hint.

  • $\begingroup$ Thank you. So, how is PCA analysis helpful to determine best contributing variables? Or is it used for a different purpose? $\endgroup$
    – user
    Commented Jul 30, 2016 at 15:31
  • $\begingroup$ Determining best contributing variables is not PCA purpose, neither a easy task to perform hereafter. Its aim is to 1. get the direction explaining as much variance within the data as possible 2. Eventually reduce the dimension of your data by keeping the most contributing new variables (which are a combinaison of your initial one) Doc $\endgroup$
    – Igor OA
    Commented Aug 1, 2016 at 8:35

The output from PCA is not a value of each attribute. It is eigenvalues you get. So the first value is direction with the highest variance, or first component if you wish. This component is combination of attributes.

No need to repeat what have been already written, check this question.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.