Struggling with understanding RandomForest model with SMOTE

Question

From what I understand my code is telling me that my base model is performing at 96% on it's training data, 55% on it's test data. And my SMOTE model is performing at ~96% on both.

From my understanding, the SMOTE model performing 96% on it's test data implies that on any new data it is given, it should perform at around 96%. However when I introduce a brand new dataset of identical data from a different time period, it's performing significantly worse.

Is anyone able to tell me if there's something I've missed/overlooked with the code below? If not, I know to look into the new dataset I've added to look for problems.

My only possible lead at the moment is that I've used the SKLearn.preprocessing OrdinalEncoder for both the main & brand new dataset to turn continuous non-integer codes into integers, which I wonder may be causing a mis-match between datasets.

I've attached the code for the main model below.

df = df.filter(["Feature1","Feature2","Feature3","Feature4",
                "Feature5","Feature6","Feature7",
                "Feature8","TargetClassification"])

y = df["TargetClassification"].values
X = df.drop("TargetClassification",axis=1)

sm = SMOTE(random_state=42)
X_sm, y_sm = sm.fit_resample(X,y)

XB_train, XB_test, yB_train, yB_test = train_test_split(X,y,train_size=0.7)
XS_train, XS_test, yS_train, yS_test = train_test_split(X_sm,y_sm,train_size=0.7)

my_SMOTE_model = RandomForestClassifier(n_estimators=100,criterion="gini",random_state=1,max_features=4)
my_BASE_model = RandomForestClassifier(n_estimators=100,criterion="gini",random_state=1,max_features=4)

my_BASE_model.fit(XB_train,yB_train)
y_pred = my_BASE_model.predict(X)
BASE_train_acc = round(my_BASE_model.score(XB_train, yB_train)*100,2)
print(f"Base model training accuracy: {BASE_train_acc}")


my_SMOTE_model.fit(X_sm,y_sm)
y_sm_pred = my_SMOTE_model.predict(X_sm)
SMOTE_train_acc = round(my_SMOTE_model.score(XS_train,yS_train)*100,2)
print(f"SMOTE model training accuracy: {SMOTE_train_acc}")
# Prints Base as 96.05, SMOTE as 96.38

yB_test_prediction = my_BASE_model.predict(XB_test)
yS_test_prediction = my_SMOTE_model.predict(XS_test)
BASE_test_acc = accuracy_score(yB_test,yB_test_prediction)
SMOTE_test_acc = accuracy_score(yS_test,yS_test_prediction)
print(f"Base model test accuracy: {BASE_test_acc}")
print(f"SMOTE model test accuracy: {SMOTE_test_acc}")
#Prints Base as 54.9%, SMOTE as 96.5%

Thank you for any help

Answer 1

Using different ordinal encoders is certainly not good, but you've also made the error of applying SMOTE before the train-test split ([1], [2]), making the test score optimistically biased. Also, accuracy is not a great metric, especially in imbalanced settings. Finally, "identical data from a different time period" may well display significantly different relationship between the independent and dependent variable, so some degradation is not unexpected.