1
$\begingroup$

0

For the past few hours I've been trying to search what this linear assumption is. Some of the articles states that that your independent variables have to be linear in relationship and need some type of transformation if there is no linearity. Other articles state that your data has to be linearly separable. Which is it? Is it both?

Does it mean that that you first have to check if the independent variables are linear in relationship, then after applying PCA, check if the data is linearly separable?

OR

Check if the data, before applying PCA, is linearly separable with techniques like linear programming.

Then there is KERNEL PCA which after searching states that it is an extension of PCA where it is applied to nonlinear data. Does that mean nonlinear in relationship or linear inseparable?

Improve this question
$\endgroup$
2
$\begingroup$

PCA is the best (in the mean-squared error sense) linear decomposition method.

PCA is defined as an orthogonal linear transformation that transforms the data to a new coordinate system such that the greatest variance by some scalar projection of the data comes to lie on the first coordinate (called the first principal component), the second greatest variance on the second coordinate, and so on. Wikipedia

The term "Linear" in PCA means:

a. That any data point is simply a linear combination of the principal components.

b. That the data matrix ($A$) can be decomposed via linear similarity transformations to diagonal matrix ($\Sigma$).

Ie

$$A = U \Sigma U^T$$

or

$$AU = U \Sigma$$

$\Sigma$ is the diagonal matrix of variances for each basis vector.

One can see at once that the linear algebra of the above formula makes clear the meaning of Linearity in PCA.

On the other hand decomposition methods like ICA (Independent Component Analysis) cannot be expressed via linear algebra as PCA above, since they require not only decorrelated components but independent components which is a stronger condition requiring non-linearities.

See also: https://datascience.stackexchange.com/a/80361/100269

Improve this answer
$\endgroup$
7
  • $\begingroup$ What I'm asking is not why PCA is a linear transformer, but what the ASSUMPTIONS is before applying PCA. According to my searches, one of the assumptions before applying PCA is that your data has to be LINEARLY RELATED. What I'm unclear is if this means if they're talking about the relationship of the data or they're talking about linear separability. I'm not addressing the linear in PCA. Are you saying that there is no assumption about the data before applying PCA? $\endgroup$ – Jiaming He Apr 29 at 16:39
  • $\begingroup$ I agree there are no assumptions to PCA. $\endgroup$ – Jayaram Iyer Apr 29 at 16:59
  • $\begingroup$ PCA can be used on any data, and either it will produce good decompositions or it will not. There is no real way to check these "assumptions" on unknown data, but decomposition still works in the mean square error sense. It is true that if data are linear combinations of some principal components then PCA will find them, but PCA can still be used in a minimum error sense $\endgroup$ – Nikos M. Apr 29 at 17:16
  • $\begingroup$ As a linear transformation (that decorrelates the data) PCA can be used on any data, whether it is useful is another matter $\endgroup$ – Nikos M. Apr 29 at 17:18
  • $\begingroup$ @JiamingHe, see the comments posted $\endgroup$ – Nikos M. Apr 29 at 18:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.