# What stopping metric to chose to optimize 'sensitivity' for a GBM in H2O?

I am predicting a disease and want to get the highest possible sensitivity score for the predicted values on my validation and test set.

What stopping metric can be used to optimize the sensitivity score on validation set?

I have about 400 observations. The response variable is binary (0/1) and I have 40 predictor variables.

My current set-up uses AUC as stopping metric.

df <- as.h2o(df)
split <- h2o.splitFrame(data=df, ratios=c(0.6, 0.2)) # split 60, 20, 20%
train <- h2o.assign(split[[1]], "train.hex") # 60%
valid <- h2o.assign(split[[2]], "valid.hex") # 20%
test <- h2o.assign(split[[3]], "test.hex") # 20%

x <- setdiff(names(df), "disease")
y <- "disease"

gbm <- h2o.gbm(
x = x,
y = y,
training_frame = train,
validation_frame = valid,
ntrees = 10000,
learn_rate=0.01,
# Stopping parameters
stopping_rounds = 5, stopping_tolerance = 1e-4, stopping_metric = "AUC",
sample_rate = 0.8,
col_sample_rate = 0.8,
seed = 1234,
nfolds = 50,
score_tree_interval = 10
)
h2o.auc(h2o.performance(gbm, valid = TRUE))

• I'm not familiar with H2O, but can you use the F1 score? Commented Nov 30, 2017 at 15:07
• @Jurgy The F1 score is based on the recall & specificity. I'm, at this, only interested in optimizing the recall(=sensitivity). Commented Nov 30, 2017 at 15:45
• Optimizing for just sensitivity is a bad idea because an always positive approach will lead to maximum sensitivity: S = TP / (TP + FN), with always positive FN = 0 so you get S = TP / TP = 1. Commented Nov 30, 2017 at 15:51

One solution to make the recall more important whitout getting to the problem mentionned by @Jurgy is to use $$F_\beta$$, a modified version of $$F_1$$ where recall is considered $$\beta$$ time more important. As seen here : https://en.wikipedia.org/wiki/F1_score, $$F_\beta$$ can be formulated both in terms of recall/precision, and in term of type I /type II error. You then need to select a $$\beta$$ based on how much recall is more important to you... this can be done by considerations on the costs of type I vs the cost of type II errors.