Deciding on the number of components in PCA

I have been running my model several times now. Each time i get different results based on what number i put in my PCA component number range (I used raw numbers in the code instead of the range function).

If i put the range from 1 to the max_number (e.g.100) of components i get certain accuracy, lets say 60%, and the component number chosen is 80. So 60% at 80 components.

Now if i repeat the run with a range from 1 to 79, i get accuracy 62%, with number of component chosen as 45

If i run the whole thing again, while choosing range from 1 to 100 separated by 10 (instead of 5, or 1), e.g. range(1, 100, 10), I get a different accuracy as well.

The accuracy is varying and not linear, meaning that if the number of components increase, the accuracy will not necessarily increase.

So what should i do?

Should i run the analysis with component range 1 to max separated by 1 (e.g. range (1,max), and then each time i get a chosen component number i should investigate the series below it? Can someone help please?

Here is my code

# Search for the best combination of PCA truncation
# and class reg (LogReg).

from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression(random_state=42, class_weight= 'balanced', max_iter=5000)
pipe_logreg = Pipeline(steps=[('pca', pca), ('logreg', logreg)])

# Parameters of pipelines can be set using ‘__’ separated parameter names:
parameters_logreg = [{'pca__n_components': [1, 6, 11, 16, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100]},
{'logreg__C':[0.5, 1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500],
'logreg__penalty':['l2'],
'logreg__warm_start':['False', 'True'],
'logreg__solver': ['newton-cg', 'lbfgs', 'sag'],
'logreg__multi_class': ['ovr', 'multinomial', 'auto']},
{'logreg__C':[0.5, 1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500],
'logreg__penalty':['l1'],
'logreg__warm_start':['False', 'True'],
'logreg__solver': ['liblinear', 'saga'],
'logreg__multi_class': ['ovr', 'auto'],
}]

clflogreg = GridSearchCV(pipe_logreg, param_grid =parameters_logreg, iid=False, cv=10,
return_train_score=False)
clflogreg.fit(X_balanced, y_balanced)

# Plot the PCA spectrum (logreg)
pca.fit(X_balanced)

fig1, (ax0, ax1) = plt.subplots(nrows=2, sharex=True, figsize=(6, 6)) #(I added 1 to fig)
ax0.plot(pca.explained_variance_ratio_, linewidth=2)
ax0.set_ylabel('PCA explained variance')

ax0.axvline(clflogreg.best_estimator_.named_steps['pca'].n_components,
linestyle=':', label='n_components chosen')
ax0.legend(prop=dict(size=12))

# For each number of components, find the best classifier results
results_logreg = pd.DataFrame(clflogreg.cv_results_) #(Added _logreg to all variable def)
components_col_logreg = 'param_pca__n_components'
best_clfs_logreg = results_logreg.groupby(components_col_logreg).apply(
lambda g: g.nlargest(1, 'mean_test_score'))

best_clfs_logreg.plot(x=components_col_logreg, y='mean_test_score', yerr='std_test_score',
legend=False, ax=ax1)
ax1.set_ylabel('Classification accuracy (val)')
ax1.set_xlabel('n_components')

plt.tight_layout()
plt.show()

• What is your motivation for using PCA? I don't know if you are aware that doing PCA can actually worsen model accuracy, because you may be throwing away useful features. There are several discussions on this topic on the Cross Validated site: stats.stackexchange.com/q/52773/241093, stats.stackexchange.com/q/303602/241093, stats.stackexchange.com/q/244677/241093 Commented May 2, 2019 at 5:41
• Thank you very much for your reply Alex. I checked all 3 links, but non of them answer my question. My initial motivation is dimentionality reduction and using the principal components for doing the classification (as i don't know which features are more important than others to classify). If you notice from my Q, i included from 1 to max feature number, so if the max features indeed give the best accuracy, then it would be used! Commented May 2, 2019 at 6:24
• My recommendation would be to plot classification error for different numbers of PCs and for other values within your grid search, so you can see how the error varies as the number of PCs increases. As you already said, you will probably not see a linear relationship, but can see what the optimal number should be. Commented May 2, 2019 at 6:51

The accuracy is varying and not linear, meaning that if the number of components increase, the accuracy will not necessarily increase.

This is not unexpected. Choosing the right number of components requires balancing the extra information given by the additional dimensions and the useless noise and redundancy present therein. Even though PCA is a linear transform, and even if you used a linear classifier, the dependence of the classifier performance on the number of components is generally very non-linear.

Two options (or alterations thereof) are common:

1. Pick a predefined level of variance to keep (usually 90% or 95%, at least for me) and just stick with it. Think of it as regularization, rather than as a hyper-parameter.

2. Treat it as a hyper-parameter and optimize over a validation set for the number of components to keep $$k$$. This is essentially what you are doing already. It's perfectly reasonable to simply choose the $$k$$ that gave the best performance (on a held-out set).

• If you have no prior Knowledge of the subject matter (you don't recognise the meaning behind the principal components), then as you suggest option 2 is the best. The code presented however doesn't include a held out set which is fundamental to ensuring the output is useful. Commented Jul 17, 2019 at 20:37
• @fswings Very true. Briefly looking at the code, it seems to use 10-fold cross-validation, which would be acceptable for choosing $k$. Commented Jul 17, 2019 at 22:08
• Take your original data and use train_test_split to produce a hold out set and the remainder. Use the remainder for the GridSearchCV to get the best model. Once identified apply it against the hold out set as a sanity check. Commented Jul 18, 2019 at 19:22
• @tsumaranaina I think what fswings is saying is that you should use a held-out test set, i.e. three splits, not two: train, validate, and test. Commented Jul 18, 2019 at 20:05
• @tsumaranaina Check out stats.stackexchange.com/questions/9357/… and stats.stackexchange.com/questions/19048/… :) Commented Jul 19, 2019 at 3:54