4
$\begingroup$

I am quite new to machine learning and python as well. I faced an imbalanced dataset and wanna use cross validation and oversamopling like the figure shown.

enter image description here

I realised the Python function below cannot be directly used for this purpose and please advice some codes for this task.

cross_val_score(model, X_train,np.ravel(y_train), cv=n_folds, n_jobs=1, scoring='roc_auc')
$\endgroup$
1
$\begingroup$

Adding to what Himanshu Rai said, you should be careful not to oversample before the StratifiedKFold, once you risk to put the same sample in both the training and testing folds check this, (where you took your image from) and that doesn't really evaluate your model's capacity to data that it never saw. What i did was to use StratifiedKFold, and then oversample or SMOTE (or whatever you want) all the training folds separately, by only looking at that fold's training data. Then use all those folds to validate your models.

$\endgroup$
1
$\begingroup$

Stratified K fold is not the answer here.

An example of code creating an oversampling k-fold class for this purpose:

class oversampled_Kfold():
    def __init__(self, n_splits, n_repeats=1):
        self.n_splits = n_splits
        self.n_repeats = n_repeats

    def get_n_splits(self, X, y, groups=None):
        return self.n_splits*self.n_repeats

    def split(self, X, y, groups=None):
        splits = np.split(np.random.choice(len(X), len(X),replace=False), 5)
        train, test = [], []
        for repeat in range(self.n_repeats):
            for idx in range(len(splits)):
                trainingIdx = np.delete(splits, idx)
                Xidx_r, y_r = ros.fit_resample(trainingIdx.reshape((-1,1)), 
y[trainingIdx])
                train.append(Xidx_r.flatten())
                test.append(splits[idx])
        return list(zip(train, test))
...
...
rkf_search = oversampled_Kfold(n_splits=5, n_repeats=2)
...
output = cross_validate(clf,x,y, scoring=metrics,cv=rkf)

Where ros was the Random oversampler from imblearn.

$\endgroup$
-1
$\begingroup$

Use stratified K-Fold cross validation, it tries to balance the number of positive and negative classses for each fold. Kindly look here for the documentation and examples. If it still doesnt solve your problem of imbalance please look into SMOTE algorithm, here is a scikit learn implementation of it.

$\endgroup$
  • $\begingroup$ I don't think stratified K-Fold does what you say (you seem to imply that it performs oversampling, which is what OP wants). Reading the source code for sklearn function you link, it says in a comment: "The folds are made by preserving the percentage of samples for each class." $\endgroup$ – Jeremy Lane May 12 '18 at 17:30
  • 1
    $\begingroup$ To clarify what I mean, your statement "it tries to balance the number of positive and negative classes for each fold" is false. $\endgroup$ – Jeremy Lane May 12 '18 at 17:46
  • 1
    $\begingroup$ stratified K-Fold preserves the same ratio of positive and negative at each fold, but does not do over/undersampling. if your entire training dataset is 70% positives and 30% negatives, so each fold will have 70/30 ratio as well. $\endgroup$ – Serendipity Dec 3 '18 at 7:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.