Background
Trying to identify the number of primary components to use (k) for PCA for MNIST aiming at 95%.
from sklearn.datasets import fetch_openml
mnist = fetch_openml('mnist_784', version=1)
# Split data into training and test
X, y = mnist["data"], mnist["target"]
X_train, y_train = X[:60000], y[:60000]
COVERAGE=0.95
If I follow Coursera Machine Learning - Principal Component Analysis Algorithm it is 67.
from sklearn.preprocessing import StandardScaler
X_centered = StandardScaler().fit_transform(X_train - X_train.mean(axis=0))
covariance_matrx = X_centered.T.dot(X_centered)
U, s, Vt= sp.linalg.svd(covariance_matrx)
calculated_coverages = ((s ** 2) / (len(s) -1)).cumsum()
calculated_coverages = calculated_coverages / calculated_coverages[-1]
k = np.argmax(np.array(calculated_coverages) >= COVERAGE)
print("k-th component to cover {0} is {1}".format(calculated_coverages[k], k))
k-th component to cover 0.9507022719172283 is 66
However, if I use explained_variance_ratio_ from scikit learn, it is 154.
from sklearn.decomposition import PCA
pca = PCA()
pca.fit(X_train)
contributions = pca.explained_variance_ratio_
coverages = pca.explained_variance_ratio_.cumsum()
k = np.argmax(coverages >= COVERAGE)
print("k-th primary compoent for 95% coverage is {}".format(k + 1))
k-th primary compoent for 95% coverage is 154
When I look at scikit-learn/sklearn/decomposition/_pca.py, it looks the logic is the same.
U, S, V = linalg.svd(X, full_matrices=False)
# flip eigenvectors' sign to enforce deterministic output
U, V = svd_flip(U, V)
components_ = V
# Get variance explained by singular values
explained_variance_ = (S ** 2) / (n_samples - 1)
total_var = explained_variance_.sum()
explained_variance_ratio_ = explained_variance_ / total_var
singular_values_ = S.copy() # Store the singular values.
Question
Please help understand why they are different.
n_components
, and will keep the top k components that have this number as the ratio of explained variance (what you called "Coverage"). SKLearn docs $\endgroup$