Track underlying observation when using GridSearchCV and make_scorer

Question

I'm doing a GridSearchCV, and I've defined a custom function (called custom_scorer below) to optimize for. So the setup is like this:

gs = GridSearchCV(estimator=some_classifier,
                  param_grid=some_grid,
                  cv=5,  # for concreteness
                  scoring=make_scorer(custom_scorer))

gs.fit(training_data, training_y)

This is a binary classification. So during the grid search, for each permutation of hyperparameters, the custom score value is computed on each of the 5 left-out folds after training on the other 4 folds.

custom_scorer is a scaler-valued function with 2 inputs: an array $y$ containing ground truths (i.e., 0's and 1's), and an array $y_{pred}$ containing predicted probabilities (of being 1, the "positive" class):

def custom_scorer(y, y_pred):
    """
    (1) y contains ground truths, but only for the left-out fold
    (2) Similarly, y_pred contains predicted probabilities, but only for the left-out fold
    (3) So y, y_pred is each of length ~len(training_y)/5
    """

    return scaler_value

But suppose the scaler_value returned by custom_scorer depends not only on $y$ and $y_{pred}$, but also knowledge of which observations were assigned to the left-out fold. If I have only $y$ and $y_{pred}$ (again: the ground truths and predicted probabilities for the left-out fold, respectively) when the custom_scorer method is called, I don't know which rows belong to this fold. I need a way to track which rows of training_data get assigned to the left-out fold at the point when custom_scorer is called, e.g. the indices of the rows.

Any ideas on the easiest way to do this? Please let me know if clarification is needed. Thank you!

Answer 1

Firstly; this is a really clear, well written question. Kudos!

I think the answer is to take the folding out of the CV and do this manually. You can generate the indices of the training and testing data using KFold().split(), and iterate over them in this manner:

from sklearn.model_selection import KFold, GridSearchCV
import pandas as pd
from sklearn.ensemble import RandomForestClassifier

iris = pd.read_csv('https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv')

kf = KFold(n_splits=3)

for train_idx, test_idx in kf.split(iris):
    print(train_idx, test_idx)

And what you'll get is three sets of 2 arrays, the first being the indices of the training samples for this fold and the second being the indices of the testing samples for this fold. Using that, you could manually cross-validate like this:

kf = KFold(n_splits=3)

x = iris.drop('species', axis=1)
y = iris.species

max_depths = [5, 10, 15]

scores = []

for i in range(len(max_depths)):
    rfc = RandomForestClassifier(max_depth=max_depths[i])
    scores.append({'max_depth':max_depths[i], 'scores':[]})
    for train_idx, test_idx in kf.split(iris):
        rfc.fit(x.iloc[train_idx], y.iloc[train_idx])
        scores[i]['scores'].append(custom_scorer(y.iloc[test_idx], rfc.predict(x.iloc[test_idx]), train_idx, test_idx)

So that's running once per value in max_depths, setting that parameter to the appropriate value in a RandomForestClassifier. It's then fitting 3 times, once per fold defined in KFold() and passing several things to the call to custom_scorer()...

1. y.iloc[test_idx] which is our y_true
2. rfc.predict(x.iloc[test_idx]) which is our y_pred
3. train_idx which is the indices of the samples of our training data
4. test_idx which is the indices of the samples of our testing data

Hope that helps. Out of interest: why do you need to know which observations are left out?