I want to repeat my experiment pipeline to validate my feature extraction. My goal is with different set of data which randomly picked as train test and validation, I want to check if the result are consistent with same classifier method (ex: SVM).
Let say i have list of ECG signal record which has id label for two class (class a = [records 1,2,3,4,5,6,7,8,9,10] and class b [records 11,12,13,14,15,16,17,18,19,20]) which i have set for train and valid and testing proportion of each record is 50%:20%:30% and both a and b are randomly shuffled per my experiment test. For example: Train = a[1,3,5,10,9],b[11,20,19,17,16] Valid = a[2,4] b [14,15] Test = a[6,7,8] b[12,13,18]
For each row in class a, beside record_name, it has time_id (30 to 1). The goal is i want to loop through the time_id to validate and test the feature i used. But in class b, since it is a normal ECG signal, it is only one row per record, so the same class b rows will be used if the time is changed in test or validation process.
This is the method to validate and test (I use classification_result from sklearn):
t = 30 for tid in range (t): a_data = a.loc[a["MINUTE"]==t] test = [a_data,b_data] test_data = concat(test).values Y_true = test_data.pop("CLASS").values Y_pred = model.predict(test_data) print(classification_result(Y_pred,Y_true) tid = tid+1
Then as usual i do training validation and testing from the set of data
to make it easier, i try to loop my process which look like this:
n_exp = 100 for k in range(n_exp): a_train,a_valid,a_test = <logic to shuffle list of class A> b_train,b_valid,b_test = <logic to shuffle list of class A> a_set_train= A.loc[A['NAME']isin(a_train)] ... b_set_test= B.loc[B['NAME']isin(b_test)] train_set = [a_set_train,b_set_train] train = concat(train_set) model = SVM() model.fit(train) test_method(validation_data) test_method(test_data)
But when i try to run these sequences, it always give the same result for each experiment even with different train valid and test records. Is there any proper way to repeat these experiments using loop, since if I hardcoded or rerun the program will consume more time.