What is the actual output of Principal Component Analysis?

Question

I'm trying to understand PCA, but I don't have a machine learning background. I come from software engineering, but the literature I've tried to read so far is hard for me to digest.

As far as I understand PCA, it will take a set of datapoints from an N dimensional space and translate them to an M dimensional space, where N > M. I don't yet understand what the actual output of PCA is.

For example, take this 5 dimensional input data with values in the range [0,10):

// dimensions:
// a  b  c  d  e

[[ 4, 1, 2, 8, 8],      // component 1
 [ 3, 0, 2, 9, 8],
 [ 4, 0, 0, 9, 1],
 ...
 [ 7, 9, 1, 2, 3],      // component 2
 [ 9, 9, 0, 2, 7],
 [ 7, 8, 1, 0, 0]]

My assumption is that PCA could be used to reduce the data from 5 dimensions to, say, 1 dimension.

Data details:

There are two "components" in the data.

1. One component has mid a levels, low b and c levels, high d, and nondeterministic e levels.
2. The other component has high a and b levels, low c and d levels, and nondeterministic e levels.

This means that the two components are most differentiated by b and d, somewhat differentiated by a, and negligibly differentiated by c and e.

Outputs?

I'm making this up, but say the (non-normalized) linear combination with the highest differentiating power is something like

5*a + 10*b + 0*c + 10*d + 0*e

The above input data translated along that single axis is:

[[110],
 [105],
 [110],
 ...etc

Is that linear combination (or a vector describing it) the output of PCA? Or is the output the actual reduced dataset? Or something else entirely?

Answer 1

I agree with dpmcmlxxvi's answer that the common "output" of PCA is computing and finding the eigenvectors for the principal components and the eigenvalues for the variances, but I can't add comments yet and would still like to contribute.

Once you hit this step of calculating the eigenvectors and eigenvalues of the principal components, you can do many types of analyses depending on your needs.

I believe the "output" you are specifically asking about in your question is the resultant data set of applying a transformation or projection of the original data set into the desired linear subspace (of n-dimensions). This is taking the output of PCA and applying it on your original data set.

This PCA step by step example may help. The ultimate output of this 6 step analysis was the projection of a 3 dimensional data set into 2 dimensions. Here are the high level steps:

1. Taking the whole dataset ignoring the class labels
2. Compute the d-dimensional mean vector
3. Computing the scatter matrix (alternatively, the covariance matrix)
4. Computing eigenvectors and corresponding eigenvalues
5. Ranking and choosing k eigenvectors
6. Transforming the samples onto the new subspace

Ultimately, step 4 is the "output" since that is where the common requirements for performing PCA are fulfilled. We can make different decisions at steps 5 and 6 and produce alternative output there.

A few more possibilities:

• You could decide to project the observations with outliers removed
• Another possible outcome here would be to calculate the proportion of variance explained by one or any combination of principal components. For example, the proportion of variance explained by the first two principal components of K components is (λ1+λ2)/(λ1+λ2+. . .+λK).
• After plotting the projected observations into the first two principal components (as in the given example), you can impose a plot of the loadings of each of the original dimensions into the subspace (scaled by the standard deviation of the principal components). This way, we can see the contribution of the original dimensions (in your case a - e) to principal component 1 and 2. The biplot is another common product of PCA.

Answer 2

Have you tried reading the Intuition section of the PCA page on wiki? Also, the Further Considerations section, I think, explains what the various components represent and addresses your questions.

A short summary of what those sections describe is that the original components (i.e., axes) of the data do not properly represent the relationship inherit in the data. Instead, the components can be combined to yield a new set of components, or axes, that better describe how the data is distributed.

The wiki link states that "PCA can be thought of as fitting an n-dimensional ellipsoid to the data". In that line of thinking, the output of PCA is the eigenvectors and eigenvalues that define the orientation and length of the ellipsoid that best fits the data.

What you do with that ellipsoid is up to you and your application of PCA.