I wonder how can I give weight to my feature before employing PCA. I mean somehow weighted PCA. Because I know that one of the features is better than others and want to give importance to it in creating components (It is not possible to select only that feature. I should have others impact too)


After standardizing your data you can multiply the features with weights to assign weights before the principal component analysis. Giving higher weights means the variance within the feature goes up, which makes it more important.

Standardizing (mean 0 and variance 1) is important for PCA because it is looking for a new orthogonal basis where the origin stays the same, so having your data centered around the origin is good. The first principal component is the direction with the most variance, by scaling a certain feature with a weight of more than 1 will increase the variance on this axis and thus give more weight to this axis, pulling the first principal component in it's direction.

  • $\begingroup$ Thank you Jan. I have already normalized my data (0-1). Now, you mean if I multiply my feature values with say 0.7, is somehow giving importance to it? Actually increasing variance is equal to giving weight to PCA? $\endgroup$ – Arkan Jun 20 '16 at 9:38
  • 1
    $\begingroup$ First of all you should standardize it (mean = 0, variance = 1) due to the nature of PCA, it only makes sense if your data is centered around the origin. I'll explain a bit more in an edit $\endgroup$ – Jan van der Vegt Jun 20 '16 at 9:46

PCA is unsupervised method for finding the most important components. I don't see a reason why you should want add a weight. If you know what features are important, why use PCA at all? Or perform PCA on the features where you are unsure about the importance.

Further, components are created in directions with highest variance and the importance is measured by eigenvalues. So I can imagine you can somehow increase the variance (like Jan van der Vegt proposed). But that's a sorcery with very questionable output.

  • $\begingroup$ Thank you. Yes you are right but my problem set and application is a bit different. I have labels and know the importance of this feature but I should deploy a unsupervised method. $\endgroup$ – Arkan Jun 20 '16 at 9:59
  • $\begingroup$ @HonzaB, PCA is an unsupervised feature selection algorithm. It reduces dimensions by finding redundancies in the data. There are supervised methods of feature selection which reduce dimensions by finding which variables relate better to the data. I think his idea is an ingenuous approach to introduce some of the advantages of supervised feature selection into PCA. $\endgroup$ – Ricardo Cruz Jun 27 '16 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.