Is SVD non-linear while PCA (by eigendecompostion) is linear?

Question

I am quite confused because a colleague of mine recently told me that he preferred using SVD instead of PCA (by eigendecomposition) because, contrary to the latter, the former is non-linear so it can identify also some non-linear patterns.

However, I cannot see exactly in what way SVD is non-linear since I have the impression that it simply applies a series of linear matrix multiplications (see also this StackExchange answer).

I know that t-SNE is certainly non-linear and for this reason it is sometimes called as non-linear PCA.

Is SVD non-linear while PCA (by eigendecompostion) is linear?

Answer 1

To the best of my knowledge no.

SVD and PCA are both linear dimensionality reduction algorithms. Some nonlinear dimensionality reduction algorithms are e.g. LLE, Kernel-PCA, Isomap, etc.

About t-SNE I would like to add a point. It reduces the dimensionality (and does it pretty well!) but it is only for visualization and can not be used in learning process! So be careful putting all these next to each other. In other words, they are all dimensionality reduction algorithms however, PCA and SVD can be used for feature extraction but t-SNE can not. All can be used for visualization purposes (in EDA).

I certainly recommend reading this answer. Probably the fact that "the square roots of the eigenvalues of $XX^⊤$ are the singular values of $X$" confused your friend that it's a nonlinear method.

Hope it helps. Good Luck!

Answer 2

PCA (Principle Components Analysis) is an algorithm that creates a translation+rotation matrix with the following property: the standard deviation of column zero is the highest, followed by the standard deviation of column 1, etc. Plot it, you'll see! Because of this, you can plot just columns (0,1) as (x,y) and get a moderately educational view of your dataset. This is why it is referred to as a dimensional reduction algorithm. You can, in fact, preserve all of the dimensions.

SVD (Singular Value Decomposition) is a matrix decomposition algorithm used by PCA. It is not by itself a dimensional reduction algorithm. You can use it to learn analyze your data, but not via dimensional reduction. SVD lets you rank rows and columns by how much of an outlier they are, v.s. the other rows or columns. SVD also lets you see how much entropy is in the data.