From this example taken from https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/

# define pipeline

model = DecisionTreeClassifier()
over = SMOTE(sampling_strategy=0.1, k_neighbors=k)
under = RandomUnderSampler(sampling_strategy=0.5)
steps = [('over', over), ('under', under), ('model', model)]
pipeline = Pipeline(steps=steps)

# evaluate pipeline
cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=1)
scores = cross_val_score(pipeline, X, y, scoring='roc_auc', cv=cv, n_jobs=-1)
score = mean(scores)

We see that a pipeline including over and undersampling is applied to X. My question is, is the scoring (which is computing roc_auc) performed on the original (X, y) dataset? Or on the sampled dataset? What I mean is, is the score from:

score(model(X), y)

or is it from:

score(model(undersample(oversmaple(X, y))))

Only the former makes sense to me, but given scikit-learn is scikit-learn and that it can't get "logistic regression" right, I have to ask.


1 Answer 1


I think the comment section answered it. It's doing the right thing by not applying the sampling to the test set.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.