2
$\begingroup$

I have been running my model several times now. Each time i get different results based on what number i put in my PCA component number range (I used raw numbers in the code instead of the range function).

If i put the range from 1 to the max_number (e.g.100) of components i get certain accuracy, lets say 60%, and the component number chosen is 80. So 60% at 80 components.

Now if i repeat the run with a range from 1 to 79, i get accuracy 62%, with number of component chosen as 45

If i run the whole thing again, while choosing range from 1 to 100 separated by 10 (instead of 5, or 1), e.g. range(1, 100, 10), I get a different accuracy as well.

The accuracy is varying and not linear, meaning that if the number of components increase, the accuracy will not necessarily increase.

So what should i do?

Should i run the analysis with component range 1 to max separated by 1 (e.g. range (1,max), and then each time i get a chosen component number i should investigate the series below it? Can someone help please?

Here is my code

# Search for the best combination of PCA truncation
# and class reg (LogReg).

from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression(random_state=42, class_weight= 'balanced', max_iter=5000)
pipe_logreg = Pipeline(steps=[('pca', pca), ('logreg', logreg)])


# Parameters of pipelines can be set using ‘__’ separated parameter names:
parameters_logreg = [{'pca__n_components': [1, 6, 11, 16, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100]}, 
                     {'logreg__C':[0.5, 1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500], 
                    'logreg__penalty':['l2'],
                    'logreg__warm_start':['False', 'True'],
                    'logreg__solver': ['newton-cg', 'lbfgs', 'sag'],
                    'logreg__multi_class': ['ovr', 'multinomial', 'auto']},
                     {'logreg__C':[0.5, 1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500], 
                    'logreg__penalty':['l1'],
                    'logreg__warm_start':['False', 'True'],
                    'logreg__solver': ['liblinear', 'saga'],
                    'logreg__multi_class': ['ovr', 'auto'],
                    }]

clflogreg = GridSearchCV(pipe_logreg, param_grid =parameters_logreg, iid=False, cv=10,
                      return_train_score=False)
clflogreg.fit(X_balanced, y_balanced)


# Plot the PCA spectrum (logreg)
pca.fit(X_balanced)

fig1, (ax0, ax1) = plt.subplots(nrows=2, sharex=True, figsize=(6, 6)) #(I added 1 to fig)
ax0.plot(pca.explained_variance_ratio_, linewidth=2)
ax0.set_ylabel('PCA explained variance')

ax0.axvline(clflogreg.best_estimator_.named_steps['pca'].n_components,
            linestyle=':', label='n_components chosen')
ax0.legend(prop=dict(size=12))

# For each number of components, find the best classifier results
results_logreg = pd.DataFrame(clflogreg.cv_results_) #(Added _logreg to all variable def)
components_col_logreg = 'param_pca__n_components'
best_clfs_logreg = results_logreg.groupby(components_col_logreg).apply(
    lambda g: g.nlargest(1, 'mean_test_score'))

best_clfs_logreg.plot(x=components_col_logreg, y='mean_test_score', yerr='std_test_score',
               legend=False, ax=ax1)
ax1.set_ylabel('Classification accuracy (val)')
ax1.set_xlabel('n_components')

plt.tight_layout()
plt.show()
$\endgroup$
  • $\begingroup$ What is your motivation for using PCA? I don't know if you are aware that doing PCA can actually worsen model accuracy, because you may be throwing away useful features. There are several discussions on this topic on the Cross Validated site: stats.stackexchange.com/q/52773/241093, stats.stackexchange.com/q/303602/241093, stats.stackexchange.com/q/244677/241093 $\endgroup$ – AlexK May 2 '19 at 5:41
  • $\begingroup$ Thank you very much for your reply Alex. I checked all 3 links, but non of them answer my question. My initial motivation is dimentionality reduction and using the principal components for doing the classification (as i don't know which features are more important than others to classify). If you notice from my Q, i included from 1 to max feature number, so if the max features indeed give the best accuracy, then it would be used! $\endgroup$ – tsumaranaina May 2 '19 at 6:24
  • 1
    $\begingroup$ My recommendation would be to plot classification error for different numbers of PCs and for other values within your grid search, so you can see how the error varies as the number of PCs increases. As you already said, you will probably not see a linear relationship, but can see what the optimal number should be. $\endgroup$ – AlexK May 2 '19 at 6:51
2
$\begingroup$

The accuracy is varying and not linear, meaning that if the number of components increase, the accuracy will not necessarily increase.

This is not unexpected. Choosing the right number of components requires balancing the extra information given by the additional dimensions and the useless noise and redundancy present therein. Even though PCA is a linear transform, and even if you used a linear classifier, the dependence of the classifier performance on the number of components is generally very non-linear.

Two options (or alterations thereof) are common:

  1. Pick a predefined level of variance to keep (usually 90% or 95%, at least for me) and just stick with it. Think of it as regularization, rather than as a hyper-parameter.

  2. Treat it as a hyper-parameter and optimize over a validation set for the number of components to keep $k$. This is essentially what you are doing already. It's perfectly reasonable to simply choose the $k$ that gave the best performance (on a held-out set).

| improve this answer | |
$\endgroup$
  • $\begingroup$ If you have no prior Knowledge of the subject matter (you don't recognise the meaning behind the principal components), then as you suggest option 2 is the best. The code presented however doesn't include a held out set which is fundamental to ensuring the output is useful. $\endgroup$ – fswings Jul 17 '19 at 20:37
  • $\begingroup$ @fswings Very true. Briefly looking at the code, it seems to use 10-fold cross-validation, which would be acceptable for choosing $k$. $\endgroup$ – user3658307 Jul 17 '19 at 22:08
  • 1
    $\begingroup$ Take your original data and use train_test_split to produce a hold out set and the remainder. Use the remainder for the GridSearchCV to get the best model. Once identified apply it against the hold out set as a sanity check. $\endgroup$ – fswings Jul 18 '19 at 19:22
  • 1
    $\begingroup$ @tsumaranaina I think what fswings is saying is that you should use a held-out test set, i.e. three splits, not two: train, validate, and test. $\endgroup$ – user3658307 Jul 18 '19 at 20:05
  • 1
    $\begingroup$ @tsumaranaina Check out stats.stackexchange.com/questions/9357/… and stats.stackexchange.com/questions/19048/… :) $\endgroup$ – user3658307 Jul 19 '19 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.