Deciding on the number of components in PCA

Question

I have been running my model several times now. Each time i get different results based on what number i put in my PCA component number range (I used raw numbers in the code instead of the range function).

If i put the range from 1 to the max_number (e.g.100) of components i get certain accuracy, lets say 60%, and the component number chosen is 80. So 60% at 80 components.

Now if i repeat the run with a range from 1 to 79, i get accuracy 62%, with number of component chosen as 45

If i run the whole thing again, while choosing range from 1 to 100 separated by 10 (instead of 5, or 1), e.g. range(1, 100, 10), I get a different accuracy as well.

The accuracy is varying and not linear, meaning that if the number of components increase, the accuracy will not necessarily increase.

So what should i do?

Should i run the analysis with component range 1 to max separated by 1 (e.g. range (1,max), and then each time i get a chosen component number i should investigate the series below it? Can someone help please?

Here is my code

# Search for the best combination of PCA truncation
# and class reg (LogReg).

from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression(random_state=42, class_weight= 'balanced', max_iter=5000)
pipe_logreg = Pipeline(steps=[('pca', pca), ('logreg', logreg)])


# Parameters of pipelines can be set using ‘__’ separated parameter names:
parameters_logreg = [{'pca__n_components': [1, 6, 11, 16, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100]}, 
                     {'logreg__C':[0.5, 1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500], 
                    'logreg__penalty':['l2'],
                    'logreg__warm_start':['False', 'True'],
                    'logreg__solver': ['newton-cg', 'lbfgs', 'sag'],
                    'logreg__multi_class': ['ovr', 'multinomial', 'auto']},
                     {'logreg__C':[0.5, 1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500], 
                    'logreg__penalty':['l1'],
                    'logreg__warm_start':['False', 'True'],
                    'logreg__solver': ['liblinear', 'saga'],
                    'logreg__multi_class': ['ovr', 'auto'],
                    }]

clflogreg = GridSearchCV(pipe_logreg, param_grid =parameters_logreg, iid=False, cv=10,
                      return_train_score=False)
clflogreg.fit(X_balanced, y_balanced)


# Plot the PCA spectrum (logreg)
pca.fit(X_balanced)

fig1, (ax0, ax1) = plt.subplots(nrows=2, sharex=True, figsize=(6, 6)) #(I added 1 to fig)
ax0.plot(pca.explained_variance_ratio_, linewidth=2)
ax0.set_ylabel('PCA explained variance')

ax0.axvline(clflogreg.best_estimator_.named_steps['pca'].n_components,
            linestyle=':', label='n_components chosen')
ax0.legend(prop=dict(size=12))

# For each number of components, find the best classifier results
results_logreg = pd.DataFrame(clflogreg.cv_results_) #(Added _logreg to all variable def)
components_col_logreg = 'param_pca__n_components'
best_clfs_logreg = results_logreg.groupby(components_col_logreg).apply(
    lambda g: g.nlargest(1, 'mean_test_score'))

best_clfs_logreg.plot(x=components_col_logreg, y='mean_test_score', yerr='std_test_score',
               legend=False, ax=ax1)
ax1.set_ylabel('Classification accuracy (val)')
ax1.set_xlabel('n_components')

plt.tight_layout()
plt.show()

Answer 1

The accuracy is varying and not linear, meaning that if the number of components increase, the accuracy will not necessarily increase.

This is not unexpected. Choosing the right number of components requires balancing the extra information given by the additional dimensions and the useless noise and redundancy present therein. Even though PCA is a linear transform, and even if you used a linear classifier, the dependence of the classifier performance on the number of components is generally very non-linear.

Two options (or alterations thereof) are common:

1. Pick a predefined level of variance to keep (usually 90% or 95%, at least for me) and just stick with it. Think of it as regularization, rather than as a hyper-parameter.

2. Treat it as a hyper-parameter and optimize over a validation set for the number of components to keep $k$. This is essentially what you are doing already. It's perfectly reasonable to simply choose the $k$ that gave the best performance (on a held-out set).