I have an imbalanced data set and I want to balance them with SMOTENC with cross-validation. In order to determine the performance of the classifier on the original data, I will cross-validation as follows:

Here, feature X is divided into X_train and X_val.

scoring = {'accuracy' : make_scorer(accuracy_score), 
           'precision' : make_scorer(precision_score),
           'recall' : make_scorer(recall_score), 
           'f1_score' : make_scorer(f1_score)}


results = cross_validate(estimator=model, X=X, y=Y, cv=10, scoring=scoring)

Based on this code, I can find the performance of random forest on the original data.

However, I am a bit confused about how I should compare the oversampled data with the original data. I want to determine the best proportion of data and I use the following code. Should I split the original data to train and val and oversample train data and find the performance on val data? I am wondering if the results of these codes are comparable?

percentage = np.arange(0.20,0.5,0.02)
Measurements = np.zeros( (np.size(percentage),4) )

skf = StratifiedKFold(n_splits=10)

for train_index, test_index in skf.split(X, y):
    X_train, X_val = X[train_index], X[test_index]
    y_train, y_val = y[train_index], y[test_index]

    counter = 0
    for per in percentage:

        smote_nc = SMOTENC(sampling_strategy = per, categorical_features=[0,1,2,3,4,6,8,9], random_state=0)
        X_resampled, y_resampled = smote_nc.fit_resample(X_train, y_train)

        X_resampled = pd.DataFrame(X_resampled, columns=X_val_train.columns)

        after_X_resampled = encoding(X_resampled)
        RF = RandomForestClassifier(n_estimators=50,random_state=10) 
        RF.fit(after_X_resampled, y_resampled)
        prediction = RF.predict(X_val)

        Measurements[counter,0] += accuracy_score(y_val, prediction)
        Measurements[counter,1] += precision_score(y_val, prediction)
        Measurements[counter,2] += recall_score(y_val, prediction)
        Measurements[counter,3] += f1_score(y_val, prediction)

  • $\begingroup$ In short - yes, since you have included SMOTE in your CV procedure and you apply it only to your training folds, you are OK. $\endgroup$ – desertnaut Mar 25 at 11:40