I have been running my model several times now. Each time i get different results based on what number i put in my PCA component number range (I used raw numbers in the code instead of the range function).
If i put the range from 1 to the max_number (e.g.100) of components i get certain accuracy, lets say 60%, and the component number chosen is 80. So 60% at 80 components.
Now if i repeat the run with a range from 1 to 79, i get accuracy 62%, with number of component chosen as 45
If i run the whole thing again, while choosing range from 1 to 100 separated by 10 (instead of 5, or 1), e.g. range(1, 100, 10), I get a different accuracy as well.
The accuracy is varying and not linear, meaning that if the number of components increase, the accuracy will not necessarily increase.
So what should i do?
Should i run the analysis with component range 1 to max separated by 1 (e.g. range (1,max), and then each time i get a chosen component number i should investigate the series below it? Can someone help please?
Here is my code
# Search for the best combination of PCA truncation
# and class reg (LogReg).
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression(random_state=42, class_weight= 'balanced', max_iter=5000)
pipe_logreg = Pipeline(steps=[('pca', pca), ('logreg', logreg)])
# Parameters of pipelines can be set using ‘__’ separated parameter names:
parameters_logreg = [{'pca__n_components': [1, 6, 11, 16, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100]},
{'logreg__C':[0.5, 1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500],
'logreg__penalty':['l2'],
'logreg__warm_start':['False', 'True'],
'logreg__solver': ['newton-cg', 'lbfgs', 'sag'],
'logreg__multi_class': ['ovr', 'multinomial', 'auto']},
{'logreg__C':[0.5, 1, 10, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 250, 300, 400, 500],
'logreg__penalty':['l1'],
'logreg__warm_start':['False', 'True'],
'logreg__solver': ['liblinear', 'saga'],
'logreg__multi_class': ['ovr', 'auto'],
}]
clflogreg = GridSearchCV(pipe_logreg, param_grid =parameters_logreg, iid=False, cv=10,
return_train_score=False)
clflogreg.fit(X_balanced, y_balanced)
# Plot the PCA spectrum (logreg)
pca.fit(X_balanced)
fig1, (ax0, ax1) = plt.subplots(nrows=2, sharex=True, figsize=(6, 6)) #(I added 1 to fig)
ax0.plot(pca.explained_variance_ratio_, linewidth=2)
ax0.set_ylabel('PCA explained variance')
ax0.axvline(clflogreg.best_estimator_.named_steps['pca'].n_components,
linestyle=':', label='n_components chosen')
ax0.legend(prop=dict(size=12))
# For each number of components, find the best classifier results
results_logreg = pd.DataFrame(clflogreg.cv_results_) #(Added _logreg to all variable def)
components_col_logreg = 'param_pca__n_components'
best_clfs_logreg = results_logreg.groupby(components_col_logreg).apply(
lambda g: g.nlargest(1, 'mean_test_score'))
best_clfs_logreg.plot(x=components_col_logreg, y='mean_test_score', yerr='std_test_score',
legend=False, ax=ax1)
ax1.set_ylabel('Classification accuracy (val)')
ax1.set_xlabel('n_components')
plt.tight_layout()
plt.show()