0
$\begingroup$

I have a problem that produce different training score when using pipeline and manual.

MANUAL :

#standardize data    
sc=StandardScaler()
X_train[['age','balance','duration']] = sc.fit_transform(X_train[['age','balance','duration']])
X_test[['age','balance','duration']] = sc.transform(X_test[['age','balance','duration']])

#applying SMOTE
X_oversampling , y_oversampling = over_sampling.SMOTE(random_state=42).fit_resample(X_train,y_train)

#modelling
model_lr = LogisticRegression()
model_lr.fit(X_oversampling, y_oversampling)  


#evaluation
y_pred = model_lr.predict(X_test)
y_pred_train = model_lr.predict(X_oversampling)
print(f'Train Accuracy Score : {round(accuracy_score(y_oversampling,y_pred_train),4)}')
print(f'Test Accuracy Score : {round(accuracy_score(y_test,y_pred),4)}')

#result
Train Accuracy Score : 0.835
Test Accuracy Score : 0.82

WITH PIPELINE :

pipeline_logreg = Pipeline([('sampling', over_sampling.SMOTE(random_state=42)),
                        ('logreg', LogisticRegression())])
pipeline_logreg.fit(X_train,y_train)

**the reason i dont include standard scaler in my pipeline because ive already done it 
  manually from the code above (at #standardize data code)

#evaluation
y_pred = pipeline_logreg.predict(X_test)
y_pred_train = pipeline_logreg.predict(X_train
print(f'Train Accuracy Score : {round(accuracy_score(y_train,y_pred_train),4)}')
print(f'Test Accuracy Score : {round(accuracy_score(y_test,y_pred),4)}')

#result
Train Accuracy Score : 0.8261
Test Accuracy Score : 0.82

So why the result is different on training accuracy? The test accuracy score was the same.

$\endgroup$
11
  • $\begingroup$ Welcome to DataScienceSE. There can be several reasons: in the best case, the pipeline simply prevents some overfitting. But I would be more concerned about other issues: first, is it binary classification? How many classes, are they balanced? Because accuracy could mask other problems. Also how many instances? Did you check that oversampling brings any advantage? $\endgroup$
    – Erwan
    Oct 14, 2022 at 9:39
  • $\begingroup$ @Erwan yes it's binary classification, so im trying to use logistic regression algorithm. The data is not balanced 0 : 1 = 88.7% : 11.3%. Thats why i'm doing oversampling, and oversampling sure benefit my machine learning model. Because my goal is to maximize recall metrics more than precision. If i not doing oversampling my recall score will be very low and get high precision. I'm new to data science world, so i'm learning to use other method (example: pipeline) to build my model. But turns out i got the different result (only on train score, the test result is same). $\endgroup$ Oct 14, 2022 at 10:12
  • $\begingroup$ @Erwan i know that when handling imbalanced data you should use roc_auc score to evaluate. its not wise to use accuracy. I'm also trying to use roc_auc score but it also got the different training score result when using pipeline (the test score also the same). Btw i have another question from your comment. If i'm already done oversampling for my imbalanced data should i still consider to user roc_auc score to evaluate more than accuracy? and btw my data is binary classification with two classes $\endgroup$ Oct 14, 2022 at 10:16
  • $\begingroup$ @Erwan i'm also using that method when hyperparameter tuning. the first one i'm using fit(X_oversampling, y_oversampling) to my randomized searh cv. the second method i'm using pipeline (in that pipeline i'm also using smote) and then fit(X_train, y_train). It also gave me different training score between that two methods. The result shows that its better to use pipeline because the gap between roc_auc train and roc_auc test is more small than when using first method. Which one would you choose? Sorry for many question, i'm new to the data science world and dont have someone to teach me :( $\endgroup$ Oct 14, 2022 at 10:26
  • $\begingroup$ Ok so the first issue I see is that accuracy is only 82%, and if the majority class is 88% you should obtain at least 88% (basically a naive model can obtain 88% accuracy just by predicting always the majority class). Can you show the confusion matrix, or classification report with precision/recall? That would help to understand what happens. $\endgroup$
    – Erwan
    Oct 14, 2022 at 10:53

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.