0
$\begingroup$
I have written a code for particle swarm optimization to tune the parameters of classification model. However, on every run I get different values of C and gamma.

Below is the code:

    ##Import Data
df = pd.read_excel('C:/Users/Ram Prakash/Downloads/Data.xlsx', sheet_name = 'Binary')
from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
X = df.drop('FaultType', axis = 1)
y = df['FaultType']
y = le.fit_transform(y)

from imblearn.over_sampling import SMOTE
smote = SMOTE()
X_sm, y_sm = smote.fit_resample(X, y)
#print(X_sm, y_sm)


# Train Test split
X_train, X_test, y_train, y_test = train_test_split(X_sm, y_sm, stratify = y_sm, 
test_size=0.2, random_state = None)
print(X_train.shape, X_test.shape)
scaler = StandardScaler().fit(X_train)
X_train, X_test = scaler.transform(X_train), scaler.transform(X_test)


##Classification by Default parameters
# Fit SVM classifier
clf_default = SVC(kernel='rbf')
clf_default.fit(X_train, y_train)
print('(Cross Validation) AUC Score:', np.mean(cross_val_score(estimator=clf_default, 
X=X_train, y=y_train, cv=5, scoring='roc_auc')))

# Show result
print('(Training set) Confusion Matrix:')
print(confusion_matrix(y_train, clf_default.predict(X_train)))
print('Training set accuracy', accuracy_score(y_train, clf_default.predict(X_train)))
print('(Training set) AUC Score:', roc_auc_score(y_train, clf_default.predict(X_train)))

# Show result
print('(Test set) Confusion Matrix:')
print(confusion_matrix(y_test, clf_default.predict(X_test)))
print(accuracy_score(y_test, clf_default.predict(X_test)))
print('(Test set) AUC Score:', roc_auc_score(y_test, clf_default.predict(X_test)))

##Classification of Parameters by PSO
# Set parameters
C_range = [100, 10000]
gamma_range = [0.0001, 0.1]
para = {'C': C_range, 'gamma': gamma_range}

para_pso = PSOSearchCV_SVM(estimator=SVC(kernel='rbf'), param_grid=para, 
scoring='roc_auc',cv=5, n_jobs=3, verbose=False, pso_size= 30, pso_max_iter= 30, 
velocity_scale=.5)
para_pso.fit(X_train, y_train)
print('Best parameters:', para_pso.best_param)

# Fit SVM classifier
clf_pso = SVC(kernel='rbf', C=para_pso.best_param['C'], 
gamma=para_pso.best_param['gamma'])
clf_pso.fit(X_train, y_train)
print('(Cross Validation) AUC Score:', np.mean(cross_val_score(estimator=clf_pso, 
 X=X_train, y=y_train, cv=5, scoring='roc_auc')))


# Show result
print('(Test set) Confusion Matrix:')
print(confusion_matrix(y_test, clf_pso.predict(X_test)))
print(accuracy_score(y_test, clf_pso.predict(X_test)))

Please suggest how can I get around this problem and why am I getting different values for C and gamma on every run.

$\endgroup$
6
  • $\begingroup$ What is PSOSearchCV_SVM? $\endgroup$
    – Ben Reiniger
    Commented May 4, 2023 at 19:18
  • $\begingroup$ You haven't set any random_states, so there will be natural random variation; are you observing more dramatic changes than you expect? $\endgroup$
    – Ben Reiniger
    Commented May 4, 2023 at 20:12
  • $\begingroup$ Hi @BenReiniger PSOSearchCV_SVM is a class created defined all the parameters and the PSO algorithm. Also, even when I set the random state the values are still changing, which again I cannot understand as to why this might be happening. Yes, the observations are dramatic, I get the same accuracy or fitness function value but different C and gamma values. $\endgroup$
    – Akshita
    Commented May 4, 2023 at 20:40
  • $\begingroup$ The SMOTE also has random states. $\endgroup$
    – lpounng
    Commented May 5, 2023 at 2:02
  • $\begingroup$ There are a lot of modules with random states in your code, so as general guidance I suggest checkpoining your data between each module and compare at which point they are not identical among runs. $\endgroup$
    – lpounng
    Commented May 5, 2023 at 9:38

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.