0
$\begingroup$

I am looking for some advice on how I can improve my predictive accuracy on a binary classification problem for betting on UFC (i.e., predict whether Fighter1 (usually the favorite, or Fighter2 (usually the underdog) will win. As you can tell there is a class imbalance as well as roughly 66% of Fighter1 win historically I am particularly unsure about how best to go about using classifier for borutaSHAP.

The dataset contains about 3k rows and 7k features

So current pipeline is as follows:

I delete Fighter1 and Fighter2 statistics and keep only difference and ratio stats (Fighter1 - Fighter2 stats and Fighter1/Fighter2 stats) - This reduces features size to about 3k. I also delete all fights whereby a fighter has less than 3 fights in UFC. I do decorrelation via clustering based on Spearman’s rank-order correlation - This reduces features size to about 1.2k I do a 80/20% train-test time split (no shuffling I take fights from 2010 onwards) whilst also ensuring that all previous fights of fighters fighting in upcoming fight night are included in the training set. This is where I am stuck a bit - I am evaluating different classifiers (LightGBM and XGBoost) and different optimisations techniques (e.g., Optuna) in order to then pass the leading model onto BorutaSHAP for further feature reduction. I am just not very sure if I should be optimising these classifiers and how and which parameters. I see later on big increases in training set, particularly when I really reduce feature size, however that doesn't always translate to big increase in the test set. I am selecting hyperparameters based on balanced accuracy. Is this a good thing to do? It is important that the algorithm is not swayed too much by favourites. I then pick the best classifier to be applied with borutashap and I am evaluating different thresholds (default 100% threshold is too stringent). The database with features selected by Boruta is then passed on to Autogluon with balanced accuracy. Currently I am achieving around 65% balanced accuracy on the training set and and about 59% on the test set.

My specific question is

What is the best way to select a tuned model to be sent out to boruta for the final feature selection step? Specifically, what metric should be used to tune the model, what hyperparameters are suggested and at what ranges? Keep in mind this model is not necessary being optimised for accuracy but rather selected to be used in BorutaSHAP for feature elimination. I use autogluon for binary classification later. P.S I will update here changes in accuracy from your suggestions and will give probability estimates for this Saturday's fights :)

def objective(trial):
    params = {
        'num_leaves': trial.suggest_int('num_leaves', 2, 256),
        'max_depth': trial.suggest_int('max_depth', -1, 50),
        'learning_rate': trial.suggest_loguniform('learning_rate', 0.01, 0.3),
        'n_estimators': trial.suggest_int('n_estimators', 100, 1000),
        'min_child_samples': trial.suggest_int('min_child_samples', 1, 100),
        'min_child_weight': trial.suggest_loguniform('min_child_weight', 1e-5, 1),
        'subsample': trial.suggest_discrete_uniform('subsample', 0.5, 1, 0.1),
        'colsample_bytree': trial.suggest_discrete_uniform('colsample_bytree', 0.5, 1, 0.1),
        'reg_alpha': trial.suggest_loguniform('reg_alpha', 1e-5, 10),
        'reg_lambda': trial.suggest_loguniform('reg_lambda', 1e-5, 10),
        'class_weight': trial.suggest_categorical('class_weight', [None, 'balanced']),
    }

    clf = LGBMClassifier(**params)


    return np.mean(cross_val_score(clf, X_train, y_train, cv=StratifiedKFold(n_splits=3), n_jobs=-1, scoring='balanced_accuracy'))

pruner = optuna.pruners.MedianPruner(n_warmup_steps=10)
study = optuna.create_study(direction='maximize', pruner=pruner)
study.optimize(objective, n_trials=300)

best_params = study.best_params

# Train LGBMClassifier with best hyperparameters
clf = LGBMClassifier(**best_params).fit(X_train, y_train)

#Robustness ratio analysis############################
train_error_rate = 1 - balanced_accuracy_score(y_train, clf.predict(X_train))
validation_error_rate = 1 - balanced_accuracy_score(y_test, clf.predict(X_test))

robustness_ratio = validation_error_rate / train_error_rate
######################################################
print('Robustness Ratio:', robustness_ratio)
# Select important features with BorutaShap
Feature_Selector = BorutaShap(model=clf,
                               importance_measure='shap',
                               classification=True,
                              percentile = 75)
Feature_Selector.fit(X=X_train, y=y_train, n_trials=2000, random_state=0)

# Get the features selected by BorutaShap
selected_features = Feature_Selector.Subset().columns
X_train_selected = X_train[selected_features]
X_test_selected = X_test[selected_features]

# Create new dataframes with important features only and include the target
train_data_important = X_train_selected.copy()
train_data_important['WINNER'] = y_train

test_data_important = X_test_selected.copy()
test_data_important['WINNER'] = y_test


######################################################################################################################
######################################################################################################################
metric= 'balanced_accuracy'
# Set training time limit
time_limit_s=3600*23

# Define the AutoGluon task and train the model
predictor = TabularPredictor(label='WINNER',eval_metric=metric, path=output_folder + '/winnerpred_LGB_optuna_borutashap75_balancedaccuracy_2010-latest_upcomingfight_01072023_fightnightintraindata_train-testtimesplit_v1').fit(train_data=train_data_important, presets='best_quality', time_limit=time_limit_s,calibrate_decision_threshold=True,
    excluded_model_types=['KNN'])

# Prediction
predictions = predictor.predict(test_data_important)
predictor.evaluate(test_data_important, silent=True)

# Finally, the leaderboard
leaderboard = predictor.leaderboard(test_data_important, extra_info=True)
print(leaderboard)```
$\endgroup$
1

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.