0
$\begingroup$

Based on my current understanding, the standard use of GridSearchCV scorers ( through available options such as "f1_micro") aim to maximize the average performance across validation folds. However, in my problem I want to design a custom scorer that will consider both average validation and average training performance, using the trading folds for the latter. Is there a way to achieve this e.g. with a custom scorer?

As for the motivation, this feature could be used to design a scorer that aims at a high validation performance that do not deviate too much from the training performance.

$\endgroup$

1 Answer 1

0
$\begingroup$

I think yes, you can achieve this by creating a custom scorer in scikit-learn. By default, the GridSearchCV scorer maximizes the average performance across validation folds, but you can define your own scorer to consider both the average validation and average training performance.

To create a custom scorer that combines both average validation and average training performance, you can define a function that takes the true labels, predicted labels, and model as input, and returns a score based on your desired criteria.

from sklearn.metrics import f1_score

def custom_scorer(estimator, X, y):
    # Calculate validation score (F1 score)
    y_pred = estimator.predict(X)
    validation_score = f1_score(y, y_pred, average='micro')
    
    # Calculate training score (F1 score)
    train_predictions = estimator.predict(X_train)
    training_score = f1_score(y_train, train_predictions, average='micro')
    
    # Combine validation and training scores (average)
    combined_score = (validation_score + training_score) / 2
    
    return combined_score

In this example, we use the F1 score as the performance metric, but you can replace it with any other metric that suits your needs. The model argument represents the current model being evaluated in the GridSearchCV cross-validation loop.

Once you have defined your custom scorer, you can pass it to the scoring parameter of GridSearchCV when initializing it:

from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC

# Create the grid search object with your custom scorer
grid_search = GridSearchCV(estimator=SVC(), param_grid=param_grid, scoring=custom_scorer)

grid_search.fit(X, y)

By using your custom scorer, the GridSearchCV will optimize the hyperparameters based on the combined score of both average validation and average training performance.

$\endgroup$
2
  • 1
    $\begingroup$ Ok, but how is X_train and y_train passed to the customer scorer function? They are not in the function signature. $\endgroup$
    – Enk9456
    Jun 11, 2023 at 3:43
  • $\begingroup$ @Enk9456 You have to split dataset into train and test then X_train is define from global scope. note: you can pass X_train and y_train into fit method of gridsearch. $\endgroup$
    – Moein
    Jun 12, 2023 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.