2
$\begingroup$

I am trying to understand why eVec (produced by np.linalg.eig) is different than pca.components_.T from the instance of the PCA class. It was my understanding that the eigenvecters of the covariance matrix are the principal components after descending sort by eigenvalues.

An explanation in simple terms would be appreciated.

import pandas as pd
import numpy as np
from sklearn.decomposition import PCA
from sklearn import preprocessing
from sklearn.preprocessing import StandardScaler

df = pd.read_csv(
'https://gist.githubusercontent.com/seankross/a412dfbd88b3db70b74b/raw/5f23f993cd87c283ce766e7ac6b329ee7cc2e1d1/mtcars.csv'
)

df = df.drop(['model', 'vs', 'am'], axis = 1)

df = df.apply(lambda x: pd.to_numeric(x))

M = df.to_numpy()

Mnorm = M-np.mean(M, axis=0)

Mnorm = Mnorm/np.std(M, axis=0)
#  This is the normalized source data.

C = (Mnorm.T @ Mnorm) / (Mnorm.shape[0] - 1)
#  This is the Covariance Matrix without bias.

eVal1, eVec1 = np.linalg.eig(C)

eVal = eVal1[np.flip(np.argsort(eVal1))]
#  eVal is sorted according to the order of the eigenvalues.

eVec = eVec1[np.flip(np.argsort(eVal1))]
#  The same sort order as above is applied to the eigenvectors.

### From sklearn:

scaler = StandardScaler()

scaler = scaler.fit(df.to_numpy())

Anorm = scaler.transform(df.to_numpy())   

pca = PCA(n_components=9)

pca_transform = pca.fit_transform(Anorm)

assert (Mnorm == Anorm).all().all()
#  This tests that Mnorm was probably constructed correctly.

assert (C.round(10) == pca.get_covariance().round(10)).all().all()
#  This indicates that the Covariance Matrix (C) was constructed correctly - the rounding is arbitrary.

assert (eVec.round(5) == pca.components_.T.round(5)).all().all()
#  However, eVec and pca.components_.T are not equal.
$\endgroup$
1
$\begingroup$

Two problems:

  1. Your sorting is incorrect:
eVec = eVec1[np.flip(np.argsort(eVal1))]

sorts the rows of the matrix, but you want to sort the columns. Replacing this with

eVec = eVec1[:, np.flip(np.argsort(eVal1))]

fixes this issue.

  1. The sign of the eigenvectors are sometimes opposite. (That's fine, being an eigenvector is scale invariant, and while both np.linalg and sklearn use unit eigenvectors, that still leaves a choice of two. I'm not sure how the packages end up picking one.)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.