Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I've understood that principal component analysis is a dimensionality reduction technique i.e. given 10 input features, it will produce a smaller number of independent features that are orthogonal and linear transformation of original features.
Is PCA by itself considered as a learning algorithm or it is a data pre-processing step.
It's not uncommon for someone to label it as an unsupervised technique. You can do some analysis on the eigenvectors and that help explain behavior of the data. Naturally if your transformation still has a lot of features, then this process can be pretty hard. Nevertheless it's possible thus I consider it machine learning.
Edit:
Since my answer was selected (no idea why) I figured i'll add more detals.
PCA does two things which are equivalent. First, and what is commonly referred, it maximizes the variances. Secondly, it minimizes the reconstruction error by looking at pair-wised distances.
By looking at the eigenvectors and eigenvalues, it becomes rather simple to deduce which variables and features are contributing to the variance and also how different variables move in conjunction with others.
In the end, it really depends on how you define "learning". PCA learns a new feature space that captures the characteristics of the original space. I tend to think that can be meaningful.
Is it complex? No, not really, but does that diminish it as an algorithm? No I don't think so.
PCA is actually just a rotation. Seriously, that's all: it's a clever way to spin the data around onto a new basis. This basis has properties that make it useful as a pre-processing step for several procedures.
The basis is orthonormal. This is incredibly useful if your features exhibit multicolinearity (two or more features are linearly dependent): applying PCA is guaranteed to give you a basis where this is no longer a problem. This procedure is known as principal component regression
The basis vectors are meaningful with respect to the spread of the data: they are the eigenvectors of the covariance matrix. This second property gives rise to PCA's famous utility as a dimensionality reduction technique: after rotating the data, projecting the data onto a subset of the basis vectors associated with a significant portion of the total variance yields a lower dimensional representation that (often) retains (most of) the (interesting) structural properties of the data.
So: is it a learning algorithm? This is sort of a philosophical question. What makes something a learning algorithm? Certainly PCA isn't a "supervised" learning algorithm since we can do it with or without a target variable, and we generally associate "unsupervised" techniques with clustering.
Yes, PCA is a preprocessing procedure. But before you write it off completely as not "learning" something, I'd like you to consider the following: PCA can be calculated by literally taking the eigenvectors of the covariance matrix, but this is not how it's generally done in practice. A numerically equivalent and more computationally efficient procedure is to just take the SVD of the data. Therefore, PCA is just a specific application of SVD, so asking if PCA is a learning algorithm is really asking if SVD is a learning algorithm.
Now, although you may feel comfortable writing off PCA as not a learning algorithm, here's why you should be less comfortable doing the same with SVD: it is a surprisingly powerful method for topic modeling and collaborative filtering. The properties of SVD that make it useful for these applications are exactly the same properties that make it useful for dimensionality reduction (i.e. PCA).
SVD is a generalization of the eigendecomposition, and that too is extremely powerful even as a constrained version of SVD. You can perform community detection on a graph by looking at the eigenvectors of the adjacency matrix, or determine the steady-state probabilities of a markov model by looking at the eigenvectors of the transition matrix, which coincidentally is also essentially how PageRank is calculated.
Under the hood, PCA is performing a simple linear algebra operation. But, this is exactly the same operation that underlies a lot of applications that most people wouldn't question applying the label "machine learning" to. This class of algorithms is called Matrix Factorization, and even extends to sophisticated techniques like word2vec: indeed, you can actually get word2vec-like results by literally just applying PCA to a word co-occrrence matrix. Generalizing again, another word for the results of PCA is an embedding. Word2vec is probably the most famous example of an embedding, but constructing embeddings (as intermediaries) is also an important component of the encoder-decoder architecture used in RNNs and GANs, which are the bleeding edge of ML research right now.
So back to your question: is PCA a "machine learning algorithm?" Well, if it's not, you should be prepared to say the same about collaborative filtering, topic modeling, community detection, network centrality, and embedding models as well.
Just because it's simple linear algebra doesn't mean it isn't also magic.
Absolutely, it is not a learning algorithm, as you do not learn anything in PCA. However, it can be used in different learning algorithms to reach a better performance in real, likes the most of the other dimension reduction methods.
PCA is used to eliminate redundant features. It finds directions which data is highly distributed in. It does not care about the labels of the data, because it is a projections which represents data in least-square sense. Multiple Discriminant Analysis, MDA try to find projections which best separates the data. The latter considers the label and finds directions that data can be separated the best, although it has some details about the kind of decision that finds. To wrap up, PCA is not a learning algorithm. It just tries to find directions which data are highly distributed in order to eliminate correlated features. Similar approaches like MDA try to find directions in order to classify the data. Although MDA is so much like PCA, but the former is used for classification, it considers the labels, but the latter is not directly used for classification.
